Apparently the word "nightmare" has only been used in the sense of "bad dream" since c. 1829. Before then the term referred to the agent causing the dreams—a mare < mera, mære 'goblin, incubus'.

What word or phrase was used earlier? I'm interested, in particular, in the meaning "bad dream" as opposed to "creature causing bad dreams" or "medical condition; pavor nocturnus". I'm interested in all time periods: Anglo-Saxon/Old English, Middle English, Early Modern English, and (modern) Modern English.

Progress

Anglo-Saxon

I found swefnes wóma 'sleep-noise' as a possibility. OEME defines this as "dream-tumult, vision", and I can find a few uses in the corpus.

Alain Pannetier suggests wódan dreáme, though this seems to have the primary or exclusive sense of "fit of madness". Dreám/dreáme seems to mean 'song, music' rather than 'dream'.

Middle English

The MED has:

dwelsinge

drēming/drēm/dream/dremme: Apparently not attested with the meaning "dream" but more like "vision" (in sleep); included nightmare-type visions or dreams.

night frai/night drede 'night fear'

incubus: Alain Pannetier suggests this was used from the 18th century on, though this suggests (to me) the agency rather than the condition. It seems to have been used as far back as 1561 in the sense of "night terror", not quite the sense I hoped for.

slēp: Typically meaning "sleep" but also meaning "dream, nightmare".

bicche doughter: Apparently used to refer to a nightmare by at least one source. The modern reflex is obvious, and suggests that this was never its primary meaning.

Ephialtes/ephialtes the more: oosterwal suggests the former and the dictionary mentions the latter. Seems to primarily refer to the cause rather than the condition.

Modern English

Rhodri suggests ill dreams for mid-19th century.

Of course from the early 19th century on nightmare became common and soon pushed all others out of use except for poetic.