Related: How can I make a multiple engagement encounter fun for my party?
For clarification; by "fun" I mean make the encounter challenging, yet not deadly, and not feel like a "rinse-and-repeat" of the same actions for an extended period of time.
As a player, I have been in a few games that involved a "gauntlet-like" fight – an encounter that consisted of fighting for a long period of time (upward of 7 rounds). These fights were fun, because they required the use of almost our entire arsenal of abilities, and health.
One of these fights was against a single enemy (with a few lower-level minions that were killed off in a round or two), that lasted for 10 rounds. Obviously, simply sticking a monster that is stronger than the entire party is not going to balance out very well*, without mitigating the damage throughout the entire party (which can break immersion, if not done properly).
An example that I have experienced that drew me in almost completely was against some kind of creature that seemed to take increased damage particular types of attacks (magic damage, for example), but was not immune to all other attacks. This meant that the monster was more focused on the caster, while the rest of the party could deal regular damage, and also had to try and keep the monster away from the caster. This forced the whole party to try and coordinate to keep the monster from simply flattening the "glass-cannon" caster.
I don't want to nerf a high-level monster, or pad a lower-level monster, so it simply takes a long time for it to die, nor do I want to just fudge dice rolls to avoid killing the party with a fully-fledged high level monster. Both of these examples, while valid, would simply turn it into a marathon, which would just drag on, and not be fun for anyone.
What things do I need to take into account when constructing an engagement that is intended to endure for a long duration (7-10 rounds), against a single creature, with the intention of not killing the party?
*For example, pitting a party of four x fifth level characters against a CR9 or above monster
Best Answer
(For what it's worth, I think a party of 5th level adventurers could mop the floor with a single CR9 monster)
From what I can tell, the crux of what you're trying to accomplish here is to have a monster with a very high effective defensive CR, while keeping the offensive CR reasonable. Such a monster will be able to last a long time against the party, but not be able to wipe them easily. But as you can tell, simply jacking up the monster's HP makes for a slog.
Split the fight into multiple encounters/stages
This tactic is a classic trick in video game RPGs: after dealing a certain amount of damage, you change the boss's form or the circumstances of the fight. This allows you to mix up the fight, and effectively turns the fight into multiple consecutive encounters hidden within a single battle.
For example, a party I play in recently fought a homebrewed boss dragon. It started with some chromatic abilities, but after we dealt enough damage to it, it changed into a dragon with all of the chromatic abilities. After that form was defeated, it turned into a shadow dragon. Afterward, the DM said that he just chose a CR-appropriate dragon, and then added not-so-powerful abilities to it for each stage. The resulting fight was both long and difficult, but not too difficult.
In my own game, I have used a boss that starts in a human form, then turns into a dragon form, then into a more powerful dragon form. Alternatively, you could have the boss retreat to a different location--perhaps one where he has traps set up or some terrain-based advantage.
One nice trick is that you can use the form change to clear otherwise debilitating effects. Almost every single-monster fight ends up being lopsided because of the imbalance in the action economy, but having your monster change forms gives you an excuse to get rid of that debilitating status effect (stun? sleep?) or allow it to survive a massive nova (a paladin dropping a ton of smites in one round of attacks, for example). At the same time, it doesn't render the players' actions completely useless, because they still made that stage of the fight easier.
Give the monster an interesting defense
If you don't want to have changing forms, you could try giving your monster interesting ways of keeping itself alive.
For example, consider the Archmage (MM 342). While he technically has access to 9th level spells, none of them directly deal damage. In fact, his highest level damage-dealing spell is the 5th level cone of cold, and all of his spells of 6th level or higher are defensive. Thus, he has a great deal of potential survivability, without being able to wipe the party in one shot (which he could easily do if he had meteor swarm instead of time stop). In my game, I had a fight where an archmage could form a huge slime armor around himself, which gave him extra tentacle attacks and a shield the players had to bypass first. Thus, while boss's damage output wasn't too high, it became an interesting puzzle for the players to pierce his defenses.
In the end, the PCs are always going to be super death-dealing machines. In order to have an extended, meaningful fight, you have to find some way to make their damage output materially useful during the fight and have some way for the boss to survive that damage output. The fastest way to a boring fight is for the players to feel like their attacks aren't making a difference, until the boss suddenly drops dead.