As a GM, how can I tell what monsters will keep attacking unconscious PCs?
I am especially wondering what happens if a party all become unconscious during combat (that is, their HP all got reduced to zero and they are dying). They still have a chance to make death saving throws, I gather, and if one rolls a 20 they would gain a hit point, and become conscious, thus potentially rejoining the fight, or healing their companions.
Mainly, is there any rule for how monster(s) usually behave in this scenario (the whole party is unconscious)? If there is not, how can a DM adjudicate if they will finish the party off or not? Note: While I understand this depends on the specific opponents and the general scenario leading to the situation I am wondering what to do when the monster description or adventure don't dictate what happens?
I notice from Enemies knocking players unconscious instead of killing them for 4e that:
It is noted that most monsters will not attack characters who are dying. The DM is free to ignore that advice if desired.
Is this still true for D&D 5e or is there some other rule or general guidance for this eventuality?
Best Answer
For me, there are a few points to consider, both out-of-character and in-game. I will split the answer in these two sections.
Out-of-Character points
First, let us consider the players and the DM, rather than the characters. The point here is that sometimes, the feelings of the people playing are more important than the in-game consistency (unless, obviously, the people playing want consistency) and you can make your decisions based on that. Note: you are one of the people playing as well, so your feelings matter too.
What kind of DM are you?
There are "Killer DMs" that want to kill your players. Also there are "kind" DMs that will pull lots of punches and even create some deus ex for the party to not be TPK'd in scenarios the characters totally should die. If you already have a favorite style, follow it. From your (now removed by edit) comments on yourself, you probably don't, so I'll keep it as if you are a versatile DM that can go along with either option.
How do your players feel about PC death?
This is something common to talk about on Session 0. Will your players feel bad if their carefully-created-with-a-lot-of-work-on-background PCs die? Will they accept it and understand that probably what caused them to die was a bad decision by them, not you being an unfair DM that wants to just kill them and did it throwing a too hard encounter at them?
Remember, the main job of the DM (and the players, but we're talking about DM'ing) is to make everyone have fun. While this might be done by being strictly fair and rational and playing it completely in-character, it also can be done by metagaming and just deciding the Red Dragon was in a happy day and didn't want to kill the party, because this is what makes everyone happier and being able to keep having fun.
About myself:
While I'm not a Killer DM, I usually warn my players that if their PCs do something too dumb (entering the dragon's lair alone while making alot of noise) their characters will die. Usually my players are fine with that, and I even pull punches sometimes, mainly if they are getting killed because I'm hitting too many 20s in the d20 and too many 6s on the d6 damage, not because they were dumb. I'm guilty of some TPKs but most of them are on the players. Except for some edge cases, I never had much trouble with players feeling wronged by deaths.
In-game points
Then again, many times we are interested in keeping the in-game consistency and being "fair" - meaning the hostile characters will do what makes sense them to do.
Why is the combat happening?
First I'll recommend an interesting reading on combats here. It's talking about dnd-4e, but it applies to 5e as well. Essentially, the hostile characters have a reason to be fighting against the party - there is some conflict we are trying to solve (through violence).
Based on the motivation behind the combat, you usually can sort out what makes sense. Bandits/thieves want to steal the characters, not kill them. A Dragon protecting his lair might just want the characters to leave, dead or alive. Wolves can be really starving and eat the characters, finishing them off.
Reminder: Although you mention "when the monster description or adventure don't dictate what happens", sometimes they do.
For example, the Goblin Ambush in Lost Mine of Phandelver explicitly states
Considerations on the "Usual"
From my experience, TPKs are usually seem as a bad thing, both by players and DMs, since they mean the game essentially ends, it's a Game Over and you have to restart with different characters with different personalities and motivations and recreate everything. Mostly we will just let the party live and punish them some other way (like looting their stuff). Note that this might be biased.
Also, usually, the hostile characters motivations aren't simply "Kill the party". Check, for example, Strahd on Curse of Strahd.
Usually the party is either being an inconvenience for the BBEG plans, invading a lair or something on these lines. In neither situation killing the party is needed - scaring them, throwing them away or something like this is enough.
So, from my experience, the answer to the dnd-4e question still applies - Most of the time, either for OoC reasons or for in-game reasons, the party won't be finished off, but punished somehow else.