Monster Deadliness: In the DMG (p.82) there is guidance about the calculating encounter deadliness. This is entirely based on the XP of the monsters encountered, and each challenge rating (CR) has a fixed number of XP associated with it. So all monsters of a given CR should be comparable in terms of deadliness.
By this logic, confronting for example a level one group with any 4 CR 1/4 monsters should be a similarily difficult encounter. Based on simulated combats, this equivalence is not observed. Some monsters are much more dangerous than their peers of the same CR. For example on CR 1 the Brown Bear was much deadlier than a Ghoul.
Some encounters can be much more deadly than their CR suggests, and result in unexpected Total Party Kill or permanent loss or mutilation of player characters.
Therefore, I am interested in advice on how to avoid creating encounters with monsters that are surprisingly more effective and deadly than their XP value would suggest according to the encounter building guidelines. Are there specific problem monsters?
The intent of this question is not the accuracy of the challenge calculations in the DMG (I think the encounters tend to be less deadly than what the DMG suggests). I am interested here primarily in how to avoid unexpected killer encounters caused by monsters that deviate from what you typically experience with a given challenge rating, the outliers.
Best Answer
1. Understand what Challenge Rating is and is not.
You're already on the right track here, as you have observed that there are some situations where Challenge Rating magnificently fails to capture the danger a monster or group of monsters represents. The first step in recognizing monsters that can lead to problematic encounters is understanding exactly what Challenge Rating does represent. Fortunately, the Dungeon Master's Guide gives thorough guidance on this in the chapter 9 section Creating a Monster.
The short version is that Challenge Rating is based almost entirely on the raw numbers - hit points, accuracy, damage, armor class, saves - with guidance for converting some monster features that aren't explicitly numeric into numeric equivalents that you can factor into the calculations.
This is all Challenge Rating is trying to do.
As you have observed, there can be monster features and encounter designs that are not fairly represented by Challenge Rating, and this is usually because one or monsters in an encounter have features that do not translate into numbers we can include in the calculation. We will examine a few concrete examples of this later.
To give a concrete "action item" here for getting toward a solution to your problem, I would first recommend: become very familiar with the "Creating a Monster" section in chapter 9. Study the moving parts of challenge Rating presented there, so that when you are designing encounters, you will be able to point out features that don't fit into that paradigm, because these features will be the first place to look for potentially problematic encounters.
2. Vanilla monsters are typically not going to be an issue.
This point follows naturally from the exposition above: if the Challenge Rating methodology accounts for all of a monster's features, don't worry about it. This is probably obvious, but still worth observing. When determining if an encounter may "punch above its weight class", don't spend too much time looking for issues with monsters that don't have any non-numeric features. If a monster just hits stuff, the Challenge Rating is probably accurate.
3. Know your player characters.
This one is really important: you need to know the characters really well. You need to have an understanding of what they are capable of and what their weaknesses are. One encounter might present an insurmountable obstacle for one party composition, and may be trivial for another.
For example, in one of my games, we recently traversed the Caves of Hunger beneath the frozen wastes of Icewind Dale. I won't give too many details because spoilers, but there is one encounter where the entire party may be subject to a powerful charm effect. Without a reliable method of dealing with this, this encounter can be incredibly difficult. However, I was playing a Twilight Domain cleric, who has a handy feature called Twlight Sanctuary:
My ability to just end the charm effect for free trivialized this encounter for us. Without this ability, we would have been in huge trouble. If you have a good awareness of what features the party has access to, you can keep those things in mind as you design encounters. You can ask yourself, "Does the party of any means of countering this?" If there is a monster feature the party has no counter to, then that is a feature you need to give more consideration to as you build your encounter.
4. Monsters with Paralyzing, Stunning, and Dominating features are automatically suspect.
These are features that you can assume without too much analysis may present trouble for the party. Both Paralyzed and Stunned inflict the Incapacitated condition, and will generally completely remove a character from the combat for at least one round. If you have designed your encounter around four characters, but one of them end up stunned for two rounds, that alone can be enough to overwhelm the remaining characters if the encounter was already going to be difficult without considering the incapacitating effects.
With this, I would challenge something stated in the question:
Brown bears are simple beasts. They scratch and bite, and as long as someone in the party can eat those hits, there won't be any surprises in terms of encounter difficulty. On the other hand, the ghoul, while weaker by-the-numbers than the brown bear, has this nifty feature:
If the ghoul manages to stick one of these to one of the characters, that character is useless for at least one round, and paralysis has the added complication:
Paralysis can make the party tank quite squishy once attacks against them have advantage and hit twice as hard. One failed save can turn an otherwise pedestrian encounter into a wipe.
Creatures with dominate X features have the same problem, but make the issue worse by giving the dominated characters actions to the enemy for at least one round.
5. Petrification should be a plot device until at least late Tier 2.
Petrification presents the same in-combat problems as Stunned and Paralyzed, but typically has the added bonus of being semi-permanent. Darth Pseudonym asks about this in their Q&A: Petrification at low levels. The problem with petrification is that the players generally won't have an immediately accessible means of curing unless they have a cleric and are at least 9th level for access to greater restoration:
Prior to 9th level, the party cannot cure petrification unless you give them a means to do so. Here's how I handled it:
If the party has greater restoration, then petrified is just another status condition that they can cure with what they have at their disposal. If they don't have it, then it is a candidate for creating perilous situations much like Stunned and Paralyzed, with the additional potential effect of changing the direction of the campaign for many sessions.
6. So what do you do with all of this? Playtesting.
This will add a good bit of time to your prep work. If you really want to know if an encounter might present the problems you ask about in the question, you have to get your player's character sheets and run the encounter yourself. I've done this a lot, because I like to run combat-heavy games. I enjoy combat, and have generally played with tables that feel the same way. Where my game preparation may lack in world building and NPC development, I make up for it in encounter design by spending time running encounters at my kitchen table with dice, minis, my players' character sheets, and my Monster Manual. This is why the other things I've talked about are really important: understanding Challenge Rating, knowing your player characters, and watching out for debilitating effects can save you a lot of time, since those things give you an idea of what encounters need playtesting, and which ones don't.