Magic Circle and Protection from Good and Evil

If the Hag always finds the party about the same time of night, they could cast either in preparation for the hag to keep the Hag out. Protection would be better, as the Hag might forget to detect magic and waste the casting on an attempt to touch someone who is already protected.

Protection from Evil and Good is only an action to cast, so it can be cast easily even when the attack has started.

[W]hat does Magic Circle even do if you cast it on an area that already contains an excluded creature?

Traps the creature inside. The spell says:

When you cast this spell, you can elect to cause its magic to operate in the reverse direction, preventing a creature of the specified type from leaving the cylinder and protecting targets outside it.

So could trap the Hag -- but the problem here is that the casting Magic Circle takes 1 minute. Unless the DM allows you to hold completion of a spell similar to a ready action -- But that isn't RAW.

Oil of Etherealness

Applying the oil takes 10 minutes, and it lasts 1 hour. It will allow you to be on the ethereal plane. If you fight and break the touch the Hag, the Haunt attack is used for the day. If you secure the Heartstone, she can't get back to the ethereal plane, and if you kill the Hag, she'll be banished back to Hell.

Turn the Hag

• Paladins of the Oath of Devotion's Turn the Unholy works when the Hags enters the material plane to attack.

• Paladins of the Oath of Ancient's Turn the Faithless doesn't require Sight, if the Fey or Fiend hears the chanting they have to flee.

• Paladins for the Oath of Vengeance can frighten a fey or fiend they see using Abjure Enemy.

Force Damage Won't Work, but Things of Force Do

A traveler on the Ethereal Plane is invisible and utterly silent to someone on the overlapped plane, and solid objects on the overlapped plane don't hamper the movement of a creature in the Border Ethereal The exceptions are certain magical effects (including anything made of magical force) and living beings.

However closer reading is that things made out of magical force hamper movement -- force damage doesn't pass through realms. This is made clear by a tweet from Jeremy Crawford:

https://twitter.com/JeremyECrawford/status/745062527118123009

No general rule causes force damage to pass from one plane of existence to another. #DnD

So while Magic Missle or Eldritch Blast won't help, the following will: - Leomond's Tiny Hut - Wall of Force - Force Cage

Wake the Sleeper Will Not Work

Originally I thought that waking the sleeper would work. I said:

Simple answer are typically missed.

Always keep one party member on watch. If someone always keeps watch (changing shifts so everyone gets full rest) they can look for party members having nightmares. Having a nightmare of terrifying images isn't a quiet thing -- ask any parent. Try putting the people with highest perception on watch at the times you're most vulnerable, like the middle of the night.

Once awake, the nightmare ends. If caught before an hour, the negative effects don't take hold. The ability only works on sleeping people on the material plane. The hags "Nightmare Haunting" is used up for for that 24 hours.

However, RAI based on this tweet from Mike Mearls, that isn't the way it works. The victim only has to be asleep to start the nightmare haunt, but waking doesn't end it.

Alarm Won't Tell You She's coming

I thought Alarm might give you some warning, but its effect appears to be limited to the material plane. (Unless you cast it while in the Ethereal?)

1. Yes -- banishment says it is "a more powerful version of dismissal", and it doesn't override the 20% chance, so that is in fact how it works.

2. Given that your question is tagged "pathfinder", by the Rules As Written there is no way that the PCs could wind up in Maztica or Dark Sun or Greyhawk (as these are all D&D settings).

A very literal-minded DM might argue that, since no destination within the Material Plane is specified, the PCs arrive at an entirely randomly chosen location, which is overwhelmingly likely to mean they show up in airless deep space, far from any sun. Let's hope they have a silent greater teleport spell ready!

A less literal-minded DM might treat the dismissal effect as though the PCs had cast plane shift -- they'd appear "5 to 500 miles (5d%)" from the place they left the Material Plane.

The Pathfinder Wiki has some words about the Pathfinder settings that officially exist on the Prime Material Plane. If you're determined that the player characters arrive on the surface of a human-livable planet, but you'd prefer that it be a randomly selected planet, your choices include Golarion, Earth, Androffa, Kasath, and, er, Carcosa.

Of course, this is your cosmology; if you'd like to rule that multiple D&D settings exist in your version of Pathfinder's Material Plane, that is your prerogative.