This seems balanced
You have done an excellent job of identifying the potential issues with your homebrew. I can see no reason why you can't use it for playtesting. It is certainly a power-positive change over the standard Way of the Kensai monk. However that tradition is typically underpowered to begin with so this bring it back into line with the other traditions.
Are there any loopholes, ambiguities or other issues in the wording of the class feature?
Feature is fairly well worded, I can only spot one potential issue. It is something you have already partly identified and would probably make a sensible ruling on the first time you encounter it. How does the damage die relate to magical weapons?
This special attack must use your Martial Arts die for damage, even if the weapon could normally use a larger damage die.
As you already identified this feature synthesizes well with magic weapons that deal additional damage. Is the intent for this to replace only the primary damage dice or all of them? For instance; if I had a Frosthammer that dealt 1d10 Bludgeoning + 1d6 Cold damage and had a Martial Arts die of 1d8 what would be my damage roll with this ability? 1d8(B) + 1d6 (Cold) or 1d8(B) + 1d8 (Cold)?
Is the modified subclass still reasonably balanced relative to other Monks in terms of combat effectiveness?
If anything this change makes this tradition more balanced than it was. Way of the Kensai is weaker than the other non-spellcasting traditions anyway. This change is likely something that should have been there all along. Potentially your concerns about magic weapons will lead to it being overpowered but that will entirely depend on the available of magic items within your campaign.
Would giving up any of the standard Kensei (or Monk) class features bring the balance back into line?
I don't think this is necessary as it is mostly already balanced. But if you do find in playtesting that it is stronger than expected any of the suggestions you make could be a good fix.
I concur with Anagkai's answer that you've probably gone too complicated, but I think I can offer some recommendations on how to simplify the features in a manner that may make them more streamlined at the table.
3rd Level
The manner that you set this up made me think about the Way of the Open Hand, which is definitely one of my favorite archetypes in the game. To that end, your goal is to take a signature feature of the Monk and improve it substantially. For Way of the Open Hand, they substantially improve Flurry of Blows, you are seeking to improve Patient Defense.
For this reason, I propose the following language for 3rd level:
Iron Body Technique
Starting when you choose this tradition at 3rd level, you can harness
your ki to improve your defenses and vitality. Whenever you use your
Patient Defense feature you gain the following additional benefits
until the start of your next turn:
- You may roll a Martial Arts die and gain a number of temporary hit points equal to the amount rolled.
- You have advantage on Strength and Constitution saving throws.
- Bludgeoning, piercing, and slashing damage is reduced by 2. This damage reduction increases by 2 when you reach certain levels in this.
class, at 8th level (4), 13th level (6), 18th level (8).
- Any critical hit against you becomes a normal hit.
These changes are focused on simply being a straightforward, easy to implement upgrade to Patient Defense. I've removed elements that grant extra attacks or improve attacks because that's beyond the theme of this monk which is exceptionally tough. I incorporated the proposed healing as temporary hit points to mitigate potential healing shenanigans using easily recoverable ki points.
6th Level
When I was reading through this, my persistent thought was, "Why don't you just upgrade the Deflect Arrows feature to allow it to be used on melee attacks?" Thus, that's what I propose with this revision:
Deflect Melee
At 6th level, your Deflect Arrows feature is improved and may be used
on melee weapon attacks as well. If you reduce the damage to 0, you
parry the attack perfectly and open your opponent up to counterattack.
You may spend 1 ki point to make a melee weapon attack as part of the
same reaction.
Overall, this change allows the Monk to simply continue using their features as before, but now with more opportunities to do so.
11th Level
A concern I had with the original phrasing is that you've used the term 'stance' in several spots, which made we wonder whether or not the 11th level features could be used in tandem with the 3rd level features. To my knowledge, 5th edition doesn't really have stance rules the way earlier editions did, but in those earlier editions you could only assume 1 stance at a time (I recall a UA for Tunnel Fighters which had some rules around a type of stance, but not necessarily rules for stances in general).
With that in mind, I considered simply just making a substantial improvement to the 3rd level feature at a cost of 3 ki points:
Beginning at 11th level, you may spend 3 ki points to enter a special
stance on your turn as a Bonus Action. The Iron Mountain Stance may be
chosen if you wish to further improve your defenses or the Clashing
River Stance may be chosen if you wish to improve your offense. You
may remain in the chosen stance for 1 minute, until you lose your
concentration (as if concentrating on a spell), until you use this
feature again, or you choose to end it as a Bonus Action.
Iron Mountain Stance
- You may gain the benefits of the the Iron Body Technique feature on the round you enter this stance. As a Bonus Action on your turn, you
may gain the benefits of Iron Body Technique without spending ki.
- You gain a +3 bonus to your AC.
- When a creature makes a weapon attack against one of your allies, you can use your Reaction to move up to half your speed toward the
attacker. If the ally is within your reach after this movement, you
become the target of the attack instead.
Note that these changes simply focus on allowing you to continually gain the benefits of your 3rd level feature for free while also tossing on a few additional perks to compensate for using your concentration. I did tweak the ally protection to make it so you needed to reach the ally, not the enemy, in order to intercept the attack. I've removed mention of other features, because the language associated with using those features is already provided in their descriptions.
Clashing River Stance [NOTE: I RECOMMEND DELETING THIS ALTOGETHER, SEE FINAL NOTES]
- You may gain the benefits of the Iron Body Technique feature on the round you enter this stance.
- You gain a +3 bonus to your attack rolls made with unarmed strikes and monk weapons.
- Your damage rolls made with unarmed strikes and monk weapons add an additional Martial Arts die.
These changes simply focus upon improving the effectiveness of your attacks. As it's a static bonus, I removed the additional boon of Advantage, I feel that you need to pick either or. Furthermore, the damage was nerfed because your original proposal allowed a monk to making damage rolls which would've been on the order of 4d8+5 per attack which is a lot when you consider how many attacks a Monk can make each turn. In the interest of not stepping on the alternative stance's boons, I only propose allowing the Monk to gain the perks of Iron Body Technique for free on the first round for the sake of preserving the benefits of the feature through the first round.
17th Level
I see what you're going for with this feature, but I've rephrased it a bit to sync up with the language proposed in this answer as well as try and clarify the order of operations for some of its elements. I did nix the element where you'd proposed to spike the Monk's damage output as it's more out of line than you might think.
Perfect Defense
- Your Deflect Melee feature is further improved and may now be used against Ranged Spell Attacks and Melee Spell Attacks.
- On your turn, you may spend 1 ki point to end an effect causing the Charmed or Frightened condition on a creature you can touch.
- As a Reaction, you can use your ki to shield yourself and your allies from harmful spells and effect. When you take no damage from an
area effect that allows a Dexterity saving throw, thanks to your
Evasion feature, you create a 5 foot radius safe zone around yourself.
Any creatures who are in this radius do not take damage from the
effect. You may spend ki to increase the size of the circle, each
point increasing the circle's radius by 5'.
Overall, I think the idea of a very tanky monk is fine, but I think trying to mix in nova damage into it runs afoul of stepping on the toes of archetypes like the Kensai or Drunken Master. To that end, were I you, I'd consider nixing the 11th level feature Clashing River Stance above altogether and just let this archetype focus on being a tank.
In my experience, Monks aren't meant to be nova damage dealers, but rather they take you down with a thousand cuts. There are traditions that buck that, but it's also their thing to buck it. You've chosen to focus on greater defense for monks, which is fine, but you shouldn't be getting damage at the same time.
Best Answer
This is too strong overall
I think your self-assessment is right. Monks are generally considered one of the weaker classes in 5e, so you can be a bit more agressive in your subclasses, but even accounting for that this is a bit too much. Let's walk through it:
First of this has a few too many features. Typical subclasses get one main feature at each of the levels. Some get more for example way of mercy gets 3 at 3rd, but then only 1 at each other tier. Here you give 2, 3, 3 and 1, that's too many, even if you consider the two on level three as one.
Second, this is adding a lot of damage. The average damage per round for most classes is about level + 7 points, and it can get to about twice that when using the highest damage classes in optimized builds.
This adds a 3 to 6 damage to each attack with Unarmed thrust, and monks get to make lots of attacks, easily 3 per round with flurry of blows, and 4 after level 5 with extra attack, adding up to 24 damage per round against many opponents. Likewise, advantage, at typical 65% to hit rates is worth about 23% more hits - for all your damage against armored opponents, not just the extra new damage, so that adds 12 straight and about 5 from better to hit for 17 extra damage. This alone, with no effort, puts you at the top tier of damage output.
On top of that you can spend ki for recurring bleed damage to your attacks (which also is a nightmare to track). All this damage scales, and on top of that at level 11, you get another 1d8 fire damage, or 2d8 if you grapple them, and monks tend to be good at grappling.
Third, on the defensive side, allowing a shield effect for +2 AC is already quite unbalancing. +2 AC can mean as much as 200% better survivability. On top of that you get to parry from level 6 on, boosting AC by another at least 3 points in the now rarer cases you might get hit. (Nevermind the free disarm on level 11 that goes along with it).
Heart Pierce seems also over the top. It is quite costly in ki, but dealing at least an expected 55 extra damage is worth that, especially as in many cases it will deal 75 or the equivalent of 150 damage, likely killing a key opponent on the spot and ending the fight right there. I tend to worry less about 17th level features, as 95% of all play happens in tier 1-3 so they rarely get played, but if you get to those levels, a failed Dex save means instadeath in many cases, and monsters with lots of hp tend to be weaker on Dex saves, too. Now you need to only damage that dragon enough to finish it off with a killing blow in half the time. And at that level you have over 16 ki, so nothing stops you from using this four times in a row.