I've just learned that the Protection fighting style is supposed to be used before any attack rolls are made, via this question: Can Protection Fighting Style Be Used After The Roll?
In my group, we've been using the (unresolved) attack roll as the trigger for Protection, and applying pseudo-disadvantage by rolling an additional d20 and taking the lower of the two rolls. When the initial roll has advantage, we just take both dice away and roll one of them again. These rolls are all made openly.
We're enjoying playing this way, but I'd like to understand how much more utility the party is getting out of it compared to the 'correct' way of using this fighting style. I'm interested in any answers which can quantify the difference (e.g. comparing wasted reactions & successful deflections), but more generally in the basic balance question: 'Would a right-minded party with an interest in optimisation always choose this modified version of the Protection fighting style over the other available fighting styles'?
An afterthought about the usefulness of this question:
I wanted to make sure that this version of protection wasn't so good that players would always choose it, because that's boring. But balance within a party is often primarily about ensuring that players aren't frustrated by another player consistently outshining them in combat. Because Protection directly aids other members of the party, making it more powerful isn't likely to directly ruin anyone's fun.
Best Answer
That's usually more balanced, except with high AC allies
The real question is: how much less damage does the party as a whole take when one member has a certain fighting style? The only defense-oriented fighting styles useful to a shield user are Defense, Protection, and Interception (TCE), so I'll also be looking at those, plus your version of Protection and an alternative.
A potentially more balanced homebrew Protection works as follows:
Without this modification, your version of Protection can turn a normal hit into a critical hit. It also may be too powerful when protecting high AC allies, as I'll discuss later.
Meeting the party
For the sake of comparison, all five party members are level 5. The main stat we care about is AC, so without further ado:
Setting the stage
The party will face off against a (nearly) Deadly encounter - two CR 5 creatures. I'll use an air elemental's Multiattack to represent a typical CR 5 creature.
For each comparison, only Main and one other party member (designated "Ally") are targeted by attacks. What about the rest of the party, though? If you're in an encounter with two typical enemies who spread their attacks across more than two targets, you've probably already won. Focus fire is king; even dumb enemies (who don't all gang up on one target) tend to each pick a target and stick to it.
Since this is a "party with an interest in optimisation", I'll assume that they have good tactical positioning. The AC 14 warlock will (try to) stand in the back and won't be targeted by as many attacks as the AC 19 fighter. They won't always succeed, though, so here's how often each party member is in a position to get hit:
Main will pick up the remaining slack, as they try to protect their allies. When we focus on their ability to protect the AC 14 squishy, Main will be targeted the other 75% of the time due to the squishy's good tactical positioning.
Kicking in the door
Let's simulate a million rounds of combat (each with four attacks) to see how much less damage Main and Ally take (combined), based on Ally's AC:
Your version of Protection is strictly better than RAW in all cases. That said, RAW Protection is also worse than Interception in all cases. Interception falls off when defending an AC 24 Ally, since it's quite likely they won't be hit at all.
Defense trails behind the rest, as Main's AC 19 isn't high enough to break bounded accuracy. However, Defense leaves your reaction free for opportunity attacks, shield, Sentinel, etc.
What if the enemies had advantage?
Interception and your version of Protection are barely affected by advantage for an AC 24 Ally, while everything else weakens noticeably. My proposed alternative drops off considerably at AC 24, bringing it more in line with Interception.
Conclusion
Without advantage, your version of Protection is strong, but probably not overpowered in general. Interception is much better at protecting low AC, while your Protection is much better at protecting high AC. If attacks at your table are only directed at AC 20+ targets, though, your version of Protection may end up overpowered.
With advantage, your version of Protection definitely protects high AC allies too well. Given how powerful super high AC already is, it becomes a must-pick for higher AC parties. If you still prefer Protection to Interception, I'd recommend adopting my proposed changes to both tone down high AC defense and to avoid "helpfully" turning a hit into a crit, while still improving on the RAW version.