Your Difficulty Calculation is Incorrect

The XP numbers represent the lower end of each type, such that:

• 0-299: Trivial (easier than easy)

• 300-599: Easy

• 600-899: Medium

• 900-1599: Hard

• 1600+ - Deadly

Your encounter was therefore hard, not deadly.

Difficulty Comes in Depth

Any single encounter, unless ridiculously deadly, will not challenge a party if they are allowed to have a long rest between every encounter. Difficulty comes when many encounters slowly whittle down the party's resources.

Further Reading

On the topic of creating and running challenging combat encounters, you may wish to read the following articles by the Angry GM (bad language warning):

• Creating Combats Part 1

• Creating Combats Part 2

• Creating Combats Part 3

• Running Combats

There is no guideline for Deadly graduating to TPK

A TPK is usually not a planned scenario, and so no guidelines are available to show how to introduce one -- even for the "good-natured" purpose that you're asking for, which is to find the upper bound limit of where the party can expect Deadly encounters without all dying off.

I've ran two Invisible Stalkers (CR 9) against three level 5 PCs, only one of which had See Invisibility. These three players had excellent stats, but despite there best efforts, all three characters died. The challenge not only was rated Deadly, they also didn't have the ability to bypass the Disadvantage on attacks from these invisible creatures. However, they actually would have survived this encounter if they had a way of seeing invisible things -- they brought both Stalkers to nearly within an inch of their life before they were killed.

I've been wanting to run a Rakshasa as well, but this monster cannot be presented to a non-magical party or to a party that lacks a spellcaster without level 6 spells or higher. It has only a CR 13, so four level 10 PCs can treat it as a Hard encounter; but when it's immune to nonmagical weapons and spells 5th level or lower, a party may not be able to deal even a single point of damage to this creature.

The above two examples show that a party can die if the DM presents them with a monster they are not equipped to deal with, even if the CR is below Deadly (Rakshasa) or way above Deadly (two Invisible Stalkers).

Below I'll show some exceptions to any TPK-inducing guidelines we might be able to provide.

Exceptional Case 1: Low CR monsters that can induce a TPK to a high level party

Some monsters have been granted a low CR, and yet can actually kill a high level party. As such, many DMs introduce them later on in the game, when the party can handle it (a "rule" I totally break). And so, on paper, encounters with these monsters may be rated as Easy, but it will actually be really, really Deadly.

The Shadow (CR 1/2) comes to mind which, on a hit, reduces the target's Strength score by 1d4. The target dies if their Strength hits 0. This means 4 hits from a Shadow will kill a Str 10 character, on average.

There's also the Intellect Devourer (CR 2), which can force a DC 12 Int Save, subtracting 3d6 from a target's Int score. If it brings the stat to 0, the target is stunned until they gain back a point of Int; so short of Greater Restoration, they're vegetables. On an average roll, an Int 10 character loses this save. Then they can do an intelligence contest with the stunned character and, if they win, the target instantly dies.

The Harpy (CR 1) is another. They approach this differently, but Luring Song is a save-or-die ability when used deviously. In enough numbers, they could lead the party into lava, off 200-foot cliffs, underwater to drown, or into a Sphere of Annihilation (this last one is a bit overkill; but so are Shadows and Intellect Devourers).

Exceptional Case 2: Low CR intelligent foes that perform a Coup de grâce

A DM who uses smart enemies will ignore the tank and head straight for the squishies -- particularly the cleric, druid, or bard, as they have access to healing and resurrection spells. They attack the unconscious character to ensure death. Once the party is down a person, the XP budget is adjusted and a Medium encounter may become Hard or Deadly. Another round of attacks, granted they survive, may allow them to kill the Wizard or Sorcerer next.

While this seems like a dick move, if the DM introduced enough support and foreshadowing for this particular foe to act in this way, the DM could kill the party in this way.

Although, this particular case, I haven't really tried before as a DM. However, I've been on the end of many combats where my character fell unconscious, and had I died then and there for a lack of healing, a TPK would have been introduced. Fortunately my party brought me back. Had the DM focus fired on me, though, there wouldn't have been enough time to heal me. In that instance, when the party and DM was already speculating this was going to be a TPK, there was most likely no other direction we were heading in than a real TPK.

Exceptional Case 3: Using RP to bypass combat with High CR foes altogether

Even the deadliest opponent, as long as they can be reasoned with, can be dealt with via rational discussion. Or trickery. Your mileage may vary.

I've prevented a whole civil war in a game of mine with a single well-placed illusion spell. I did have to roleplay the scene to bring it to a convincing resolution. But the DM was foreshadowing the war for a few weeks, and I apologized after I'd done it. But I think it shows that you can avert almost any crisis with roleplaying.