Let's assume the following:
- Rapier: 1d6 Critical Hit Range 18-20/×2
- Attack: +10
- Damage: 3d6+2 (extra dice from other feats, I believe only 1d6+2 gets applied on the crit.)
- Opponent AC: 20
- Battle Ardor: +2 to confirm a crit (warblade class ability)
What feat will give a higher average damage output per turn: Improved Critical (double the critical threat range) or Weapon Specialization (+2 damage)?
How does adjusting the chance to hit affect the outcome (attack vs. AC)?
How does adjusting the base damage affect the outcome?
Best Answer
Let:
$$\begin{align} m&\text{: chance of a miss} \\ h&\text{: chance of a hit} \\ c&\text{: chance of a critical hit} \\ D_m=0&\text{: damage on a miss} \\ D_h&\text{: damage on a hit} \\ D_c&\text{: damage on a critical hit} \\ D = mD_m+hD_h+cD_c&\text{: damage} \\ H \ge 2&\text{: the lowest number on the d20 which is a hit} \\ T \ge H&\text{: the lowest number on the d20 which is a threat} \\ C &\text{: the lowest number on the d20 which confirms a critical} \\ \end{align}$$
Then:
$$\begin{align} m&= {H-1\over 20} \\ h&= {1-m-c} \\ c&= {21-T\over 20}\times{21-C\over 20} \\ \end{align} $$
In general \$H=AC-\text{Attack mod}\$, however, this cannot be less than 2 or greater than 20. Normally, \$C=H\$ but because of Battle Ardor, in this case \$H=AC-\text{Attack mod}-2\$, again not less than 2 or greater than 20. We therefore need to consider a range of \$AC-\text{Attack mod}\$ from 2 to 22.
This spreadsheet crunches the numbers. For all of them Weapon Specialization is better, doing 4.3 more mean damage when you need a 2 to hit dropping to 0.2 when you need a 20.