Is there a probabilistic difference in player success between the two methods of fighting a Nemesis

The basic math does not change, no. 1d20+5-2 >= 15 is the same as 1d20+5 >= 15+2, and 3d6+5-2 >= 15 and 3d6+5 >= 15+2 are equal as well, in case you're wondering about those pesky curves.

However, there are five basic ways the outcome can still be different in certain cases.

• The first is if there are any rules or abilities that kick in on a modified roll result that's below (or above) a particular value. Angelo Fuchs gives the example of crits in CyberPunk when spending luck: the luck-modified roll, if greater than a certain threshold, explodes.
• The second way is if there's a rule that is activated when the DC or target number is above or below a particular threshold, like D&D 3.x's rule forbidding certain skill checks on DCs greater than 10 without ranks in that skill.
• The third, which I've never spotted in the wild, would be if there's a way to reduce or remove unfavorable DC modifiers as such; this seems unlikely, given the way DCs are usually based on inherent aspects of the task.
• The fourth is if there are rules that allow penalties to rolls to be subsumed by an existing penalty of the same type or removed by an ability; e.g., D&D 3.x applies only the largest penalty by default and allows feats to remove penalties regardless of amount.
• And the fifth, the odd one out, is if it only affects rolls and there's an ability or rule to forgo rolling, like D&D 3.x's take 10 rule.

From what I can gather what you're asking, you want to know the probabilistic difference between rolling 10d10 and 5d20. You've rightly pointed out that each roll has the same maximum and that each has a better chance at rolling their given averages. The averages are different, which you already know. They obviously have different minimums (10 vs 5), and so you want to know precisely how different the rolls are.

Dice rolls are commonly noted as "xdy", where x is the number of dice and y is the number of faces. "d" tells us we're looking at dice and acts as a delimiter.

Any time we look at two dice variations where the product of the number of dice (xdy) and the faces of the dice used (xdy) is equal between the variations, we usually want to know how they differ since the ranges are so similar. In our case, 10d10 versus 5d20 is very similar because 10*10 is equal to 5*20. The following answer can be used as an example for any comparison of dice rolls where x and y on both variations have the same product (2d10 versus 1d20, 2d6 versus 1d12, 3d8 versus 4d6 versus 2d12, etc).

Fewer Dice, More Variation

Using AnyDice.com we can calculate the probability very simply with the commands output 10d10 and output 5d20. And really that's all there is to it. The black line below represents 10d10, and the yellow line represents 5d20.

Generally speaking, when you have a greater number of smaller dice, your rolls are less "swingy". Meaning, there are better odds at rolling the "average". But, you have worse odds at rolling higher numbers. When you use fewer number of greater dice, your rolls are more "swingy", meaning you have less chance to roll the average and more chance to roll the extreme ends of the ranges.

Put another way: Look at this graph, it represents the odds that you will roll at least a given number. You can see in general it's better to roll 10d10 because you have greater odds at hitting a certain number until about 60, then 5d20 gives you better odds at hitting those values, but only slightly.

So with 5d20, you have higher odds at hitting a greater range of values, meaning that if you roll 5d20 often, you'll see more "swingy" results. But with 10d10, the odds are more stacked in the middle, meaning it should feel like you're hitting the "average" or the "middle" results more often.

Another Example

But let's simplify. Lets look at output 2d10 vs output 1d20. Same idea as 10d10 vs 5d20. With 2d10, the odds are much different than 1d20 because there are a greater number of rolls that represent the middle values (11). there's 10-1, 9-2, 8-3, 7-4, 6-5, 5-6, 4-7, 3-8, 2-9, and 1-10 representing 11. 10% of all the combinations are 11. But for higher values (20), there is only 10-10 representing that, which is only 1% of all possibilities. But for a 1d20, there is a 5% chance for every number. So 11 is represented by the same number of faces as 20, or 1.

Similarly, if you wanted to compare 1d100 to 5d20 and 10d10, you would see a flat probability: a 1% chance for each value between 1 and 100.

Conclusion, Final Thoughts

So we can see why certain combinations of damage die are used in RPGs, and more specifically D&D 5e (which you originally asked about). The more dice you can use for a given range, the more you, as a designer, can control the probable outcome of that roll. Whereas some rolls, like loot tables, rely on an equal probability of each result by using only 1 (or very few dice) such as 1d100 rolls. Simply put, if you want to design a system that uses dice, you can control probability more by adding more dice.