Suppose, as happened last night, a character intends to subdue rather than kill an opponent. She made a melee attack, hit, declared her intent to knock out. The damage roll was enough to reduce the opponent to zero and the remaining damage exceeded the opponent's maximum HP.
Now we would seem to have two rules invoked:
Instant Death. Massive damage can kill you instantly. When damage reduces you to 0 hit points and there is damage remaining, you die if the remaining damage equals or exceeds your hit point maximum. (PHB p.197)
Knocking a Creature Out. When an attacker reduces a creature to 0 hit points with a melee attack, the attacker can knock the creature out. The attacker can make this choice the instant the damage is dealt. The creature falls unconscious and is stable. (PHB p.198)
So which rule takes precedence? Or is it neither?
Best Answer
These rules have two entirely separate triggers, and if a single attack satisfies both, then so be it.
The attack reduced the creature to 0 hit points, and it was a melee attack. So you can choose to knock the creature out; it's now unconscious and stable.
However, there was leftover damage from the attack equaling or exceeding the creature's maximum hitpoints. So it dies.
The end result is that the creature is dead. If you're worried about realism, this is a perfect outcome: If you're trying to knock someone out, but you hit them hard enough to kill them, they die, regardless of your intentions.