To find out, benchmark it
When balancing a new feature, you have to benchmark it against what already exists in the game. In this case, we have to benchmark it against all the other fighting styles. Let's assume a 16 Dex and ignore smite damage (which is constant regardless of the choice of fighting style).
Finding benchmarks
Defense: +1 AC
Build 1: Equip a shield and longsword. Overall +3 AC. DPS: 1d8+3 (7.5 ave) at level 1-4. 2d8+6 (15 ave) at level 5+
Build 2: Equip a greatsword. Overall +1 AC. DPS: 2d6+3 (10 ave) at level 1-4. 4d6+6 (20 ave) at level 5+
Dueling: +2 damage (one-handed rapier)
- Equip a shield (typical Dueling style strategy). +2 AC. DPS: 1d8+5 at level 1-4. 2d8+10 at level 5+
GWF: between plus \$\frac{4}{5}\$ and \$\frac{4}{3}\$ damage (two-handed greatsword)
Feat: Great Weapon Master (typical feat "tax"). Extra attack on a crit. Optional +10 damage per swing. No AC bonus. DPS: 2d6+3 (11.9 ave, with crit) at level 1-4, 4d6+6 (23.2 ave, with crit) at level 5+
Feat: Polearm Master (typical feat "tax"). Bonus action attack with a polearm. No AC bonus. DPS: 1d10+1d4+6 (15.3 ave) at level 1-4, 4d6+6 (24.1 ave, with crit) at level 5+
Protection: 1 reaction to impose DAdv with shield. +2 AC
- Feat: Shield Master (typical feat "tax"). Equip a longsword (highest one-handed damage die). DPS: 1d8+3 at level 1-4. 2d8+6 at level 5+
TWF: +3 damage at level 1 (one-handed weapon)
- Feat: Dual Wielder (typical feat "tax"). +1 AC. Equip two rapiers. DPS: 2d8+6 at level 1-4. 3d8+9 at level 5+
Interpreting the benchmarks
When there is a feat "tax" that typically comes with the fighting style, I include it in the benchmark. Since the system allows for this synergy, having both must be in the realm of an optimized but balanced combination.
We see a trade-off mainly between AC and damage. Specifically, the benchmarks show us:
At a +3 AC bonus: DPS is 7.5 at level 1-4, 15 at level 5+
At a +2 AC bonus: DPS is 9.5 (w/o reaction) or 7.5 (with reaction) at level 1-4, 19 or 15 at level 5+
At a +1 AC bonus: DPS is 10 (no bonus action used) or 15 (bonus action used) at level 1-4, 20 or 22.5 at level 5+
Without an AC bonus: DPS is 11.9 or 15.3 at level 1-4, 23.2 or 24.1 at level 5+
When you create your new fighting style, check where it fits in the above list, and use your intuition where it doesn't quite fit.
Evaluating your Fighting Style
This is what you proposed as a homebrew:
When a creature within range of a melee weapon you are wielding attacks a target you can see (other than you), you can use your reaction to impose disadvantage on the attack roll.
This is a very versatile fighting style since it allows for any combination of weapons and shield, and might synergize with different feats. Let's examine each of those.
Build 1: Equip a shield and longsword. Feat: Shield Master. +2 AC. DPS: 1d8+3 (7.5 ave) at level 1-4. 2d8+6 (15 ave) at level 5+, reaction use
Build 2: Equip two rapiers. Feat: Dual Wielder. +1 AC. DPS: 2d8+3 (12 ave) at level 1-4. 3d8+6 (19.5) at level 5+, reaction use
Build 3: Equip a greatsword. Feat: Great Weapon Master. No bonus to AC. DPS: 2d6+3 (10.5 ave) at level 1-4. 4d6+6 (20.5 ave) at level 5+, reaction use
Build 4: Equip a halberd. Feat: Polearm Master. No bonus to AC. DPS: 1d10+1d4+6 (14 ave) at level 1-4. 2d10+1d4+9 (22.5 ave) at level 5+, reaction use
It comes out to be the same as the Protection FS when using sword-and-board, is weaker than TWF for dual wielded weapons, and is also weaker than GWF when using a greatsword or halberd.
This is good, because a FS that is too versatile risks being better than all the others. For example, if the Paladin could throw down a shield and pick up a second sword and suddenly be as good at TWF as someone with the TWF fighting style, but pick up a shield and suddenly be as good at defense as one with the Protection fighting style, then this feat would be better than either TWF or Protection.
Is it balanced?
It is very versatile, but damage-wise, it is weaker than most of the other options if it tried to copy them. But it is equivalent to the Protection FS when equipping a sword and shield. In this regard, it is stronger than Protection, such that given a choice between this homebrew and Protection, it is always better to choose this homebrew.
However, with regards to game balance, it cannot do something that any other fighting style cannot already do. On the whole, it does not break the game.
The average roll of any single die can easily be calculated by taking the the sum of all faces and then divide by the number of faces. Luckily, this always comes to 0.5 × HIGHEST_NUMBER + 0.5. So for a d8, the average will be 4 + 0.5 = 4.5 and the average for a d10 will be 5 + 0.5 = 5.5.
This means for your attacks it's
- 1d8 + 6 = 4.5 + 6 = 10.5 average
- 1d10 + 4 = 5.5 + 4 = 9.5 average
Thus, using it one-handed is superior. Using it one-handed also allows you to do other things with your other hand - such as wielding a shield, or a second weapon - making it even more superior.
Best Answer
Only once per attack (that hits)
The Dueling Fighting Style says:
Damage Rolls are defined in the Combat section, under "Damage and Healing":
So, the sum of all the dice that were rolled (plus the modifiers) is the "damage roll", and you add the +2 to that, not to each dice.
Note, however, that this is per attack, so if you get Extra Attack (Multiattack is a monster feature, not a PC feature) you can add +2 to the damage of each attack that hits.