The DMG contains recommendations for the XP values of encounters that are supposed to help you create engaging but survivable combat encounters (page 82).
They work pretty well for low- and mid-level play. Unfortunately, for higher level play (Tier 3 and Tier 4) these recommendations are, to use a technical term, crap. At higher levels these encounters typically fail to pose adequate challenges for a competent adventuring party. I can confirm this from own experience, both as a player and as a DM.
As a DM, I have no qualms to throw monsters against high-level PCs that have a higher CR than the PCs' average level and I routinely ignore any multipliers for using more than one monster (and I tend to run a relatively low-magic equipment campaign).
Our DM does the same, regularly attacking us with multiple times the XP budget of a deadly encounter. We do have good magic items, but nothing over the top, no legendary items or such; our frontline fighter does not even have a magic weapon.
However, I think they are playing with fire by pushing ever farther beyond the guideline to break us — at some point, the risk of a TPK becomes very real1.
I am therefore interested in understanding, what has worked in your campaigns as XP budgets for high level combat encounters (Tier 3, level 11-16), compared to what the DMG says? How many times more is OK in your experience? What in your experience has been too much? Have you been ignoring multipliers for multiple monsters routinely, and how did that work?
I appreciate that it would be very hard to give an exact answer for this, a lot will depend on the party composition and equipment. I am not looking for super-exact calculations, more for broad ranges.
1 In one recent encounter, our party of 3 characters of 12th level (XP budget for a "deadly" encounter per DMG: 13,500) was ambushed by an improved Hellfire Engine, Hobgoblin Warlord, 5 Hobgoblin Captains, 4 Hobgoblin Devastators, a Hobgoblin War Priest, Priest, Bard, and 30 normal Hobgoblins on fast vehicles. The naked XP value per DMG totals up to about 34,000 XP, without applying any multipliers. With multipliers for multiple monsters, it is 136,000, or more than 10 times the budget for a deadly encounter. We somehow managed to not only survive and escape, but also kill about half of the leadership and most of the rank-and-file Hobgoblins in the process. But this could have very easily, with one bad roll – resulted in the end of the campaign. The hellfire engine is removing the possibility of raising the dead.
Best Answer
Frame challenge
I'd like to offer a bit of a frame challenge.
You say:
I think your actual problem is that the DM and the players (or at least one player) are not agreeing as to what is fun.
You might not be 90 degrees off from each other, but you're some degrees off.
The combat you described does indeed sound over-the-top, but you also say:
Sounds like a success to me. Faced with overwhelming odds, you decimated enemies that were intent on not just killing you, but destroying your very souls, and you either eliminated them as a threat, escaped their area of influence, or weakened them so much you'll be able to mop them up later.
And maybe you've bagged so much XP it's time to level up.
But there is something here that the DM has done that is not-fun to you, and that's the issue, and that's the way you should approach it.
Some tables, TPKs aren't a no-fun game
At some tables, the game is played where TPKs are possible, or even likely, or common. These 3 characters die, 3 more come in. I don't think that kind of play is probably common, but it is certainly done, and within the rules. For one table a TPK is a disaster and a horrible, no-fun session, everything is ruined; another table, it's business as usual.
Also, you can certainly play a high paranoia game where those 3 characters never make a move without scouting and scrying, always hidden, and whenever confronted with a superior force, run and hide, then regroup, strategize, and attack. History is full of examples where an inferior force defeated a superior force in just this way. (Although tbf, history is also full of TPKs.)
Handling character death
Different tables handle character death differently. Some tables, characters effectively never die permanently, or perhaps even temporarily. Other tables, characters die all the time. The DM chooses how death is handled. If you don't like it, then that's a topic to discuss with the DM.
Finding an XP recommendation won't fix your problem
Because basically what you want to say to the DM is, "you're doing it wrong, here's better XP guidelines.".
But nothing you've said suggests that the GM thinks they're doing it wrong.
XP budgets don't kill parties, DM's do. A TPK can only happen because the DM lets it happen. The DM has an infinite set of tools at their disposal to prevent it. And for many players, at least the possibility of a TPK makes the game fun.
And also an XP recommendation is likely to be imprecise
There are a lot of factors that make XP guidelines for building encounters very imprecise, especially as you level up.
They don't take into account the state of the party. The party could be well-rested, or not; buffed or not; have chosen the right spells or not; have the optimal equipment for dealing with this particular encounter, or not.
They don't take into account that as the number of participants goes up, the complexity goes up. XP becomes an extremely poor predictor of outcome given the many variables such as area-of-effect, who is aiding who, who is benefited or hindered by terrain, and even can the DM (and the players) keep track of the many participants. If the XP recommendation assumes that either the DM or the players are executing perfectly, then the extent to which it accurately depicts outcome will be heavily influenced by how well the combat is executed on either side.
XP guidelines can only average swinginess, at best. As levels and CRs go up, effects and outcomes become ever more swingy. Spells and effects like disintegrate are all-or-nothing, and scoring a hit on a leader of one side or another at the right time could turn the tide, but missing will use up a 6th-level slot and more importantly, likely a character's whole turn. And outcomes become more swingy, too.
What you can do instead
There are some tools, most of which I have used many times. I can't say there aren't too many times where I just decided not to play D&D because the game didn't seem worth the candle, but it has happened.
Talk to the DM one-on-one out of session
Hopefully you're basically friends, so somehow you should be able to talk about it. You probably can't "fix" the problem in a single discussion. However, you can become more aware of each other's viewpoints, and in my experience, that helps. I find in these conversations it's usually better to go light, and not try to get it all out all at once.
Talk with the whole table, after session
In the group I play with, there are multiple games going on, and various people are players or DMs, and we're a fairly experienced and opinionated bunch. In the games I'm in we almost always have a discussion afterwards. Usually the DM says, "Aaand that's where we'll end for tonight. Before we go, questions, comments, concerns?" Sometimes it's pretty quick, other times it's lasted quite awhile.
I'm not sure a player can effectively start this conversation, I think it probably has to come from the GM. Maybe it's something you can suggest to the GM, one-on-one.
What you can do at the end of a session as a player
What I have done as a player is two-fold. Even when I don't agree I've listened very carefully to what the DM and the other players have to say about my role in what is fun and not-fun.
Secondly, I make sure that at the end of every session I usually try to throw out a few comments about what I thought was cool about the session, whether it was a fellow player's clever maneuver or roleplay, or something cool the GM did.
I have also found in group pastimes that sometimes one person bitching and moaning becomes contagious. I try to make sure that any feedback I offer does not feed into this, at the very least it's counterproductive if my observations on something being less than perfect actually leads to things being more less than perfect.
Change the game
If you aren't having fun, don't play. Suggest a different game, or run the game yourself.
Roll with it
You can also just lean into it. Miamoto Musashi taught in The Book of Five Rings that a samurai does not fear death because they are already dead.
You could decide that this is the game, and if you're having fun, who cares if everyone dies.
No one ever likes to hear "change your attitude" as a solution, but it is true that sometimes focusing on changing yourself yields better results.
Suck it up
If you're having enough fun that you don't want to quit or play a different game, if you can't change the GM, and you can't change yourself, then just suck it up and focus on the parts you like. Again, people rarely want to hear this, but it's an option.
Calibrating encounters for higher levels
As a DM
I don't weigh everything out by XP, and the GMs I play with don't either.
I try hard to make sure character death is unlikely but possible, and that a TPK can only happen if the PCs do really dumb things on top of dumb things, or purposely decide to go down fighting. I tell the players this ahead of time. We discuss it in session 0, and along the way if we need to. I find it's more fun if the players usually triumph but not always, and sometimes have real setbacks, and feel a certain sense of risk.
I design encounters by designing an encounter I think will be fun, then add or subtract monsters until it's where I want from challenging to deadly by seat-of-the-pants, then I design in some level of early warning for the PCs and make sure they can escape. A good encounter might not be winnable or might not be escapable, but it shouldn't be unwinnable AND unescapable.
I haven't had a TPK yet, although I've had a couple that came pretty close, and some heated conversations. Externally I shrug and say, "I'll keep your concerns in mind, but we all agreed at the beginning that high danger was possible." Internally I look very carefully at what I did that resulted in something that I didn't expect, so I'll know better for next time.
As a player
As a player, I can't tell you the number of times we've gotten our butts kicked, escaping by a thread, only to re-group and either wipe out whatever it was through planning, or just avoid the encounter all together. We the players have had (good-natured (mostly)) arguments that we bugged out too soon. I'm usually of the school that as soon as it starts to go pear-shaped, you bale and hit 'em from orbit, but I tend to usually be the voice of paranoia.
This approach as players has helped, but it's mostly been after- and between-session conversations, that have allowed us to play the campaign I am currently a player in, into the top ranges of Tier 3, and soon into Tier 4.