There's really a wide range of options here. For example:
- Roll "to hit" and use fixed damage.
- Roll "to hit" and damage separately.
- D&D-style: separate "to hit" and damage, but your hit roll can give you special effects like critical hits.
- Straight-up "degree of success."
- Degree of success with modifiers, like axe gives you +2 to final damage.
- Degree of success translates into damage based on other stats or a chart.
- Roll a bunch of dice and split them to determine the characteristics of an action.
Does the "Degree of Success" Approach Require More Arithmetic?
Not inherently.
In particular, if you use roll-and-count dice pools rather than roll-and-add math, then players can find the degree of success just by sorting dice side-by-side. This still takes time but isn't as sensitive to players' math skills or level of fatigue.
Does Using "Degrees of Success" Over "Hit, Then Damage" Restrict Tactical Options or Character Variety?
Not inherently.
For instance, if you want to create a niche for a big-damage weapon, it's as easy as "If you win the attack, +2s to damage." This weapon has a very different feel from a more "balanced" weapon with "+1d to attack and defense rolls." Ditto you can create a niche for a big-damage character by just writing something like "Add your Strength to damage when you hit." (Or how about "Margin of success add damage to your hit, up to your Strength number?")
One notable way that "degrees of success" changes back-and-forth damage math is that defense still benefits you even if it falls short of stopping the attack. Consider a system where we roll attack vs. parry — I roll 5 on my attack, you get 4 on your defense: in a "to hit, then damage" system, that's a straight-up failure for you and I get to roll my damage just as if we had 5 vs. 1; in a "degrees of success" system, you've turned a potential killing blow into a glancing one.
The "Realism" Question
Defining what's "realistic" is incredibly tricky here.
Smaller weapons aren't inherently more "agile," because you don't fight with a longsword by just sweeping it back and forth in wide arcs.
Real armor both deflects blows and dissipates some of the their energy. Weapons designed to defeat armor act very differently than those which aren't.
Realistically, a cunning thief with a dagger isn't going to be able to hurt an enemy in gothic plate with "fast" or even "precise" attacks. She needs to straight-up wrestle him into a position where that blade can actually do its job.
If you've chosen to use abstract hit points, your system is already privileging gameplay or genre considerations over "realism." Embrace that! Figure out how you want combat to feel and that'll drive whether you use separate or combined damage, static defense or opposed checks, &c. Just remember that a lot of what seems "realistic" based on other media is just a set of narrative conventions.
I Want Fast Resolution, What Should I Do?
I don't think there's a major difference between separate damage rolls or degree of success, at least compared to other factors like how many modifiers are involved or the size of a dice pool.
Really, though, if you want the fastest resolution of combat, decrease the number of actions involved overall (one opposed die roll with ten situational modifiers is still going to be faster than an entire combat encounter in D&D3/D&D4), and reduce decision points that are exceedingly opaque or offer only trivial benefits.
Best Answer
I have done some game-design work, mostly in D&D 3.5e-derived settings, which are roll-over. However, some roll-under ideas were considered—albeit briefly—for some of these mechanics, so I’ve given this some thought and discussed these with other designers. I cannot claim as much experience with this question, though, as someone who has done a lot of work with roll-over, roll-under, and mixed systems. But from my perspective,
There is no particular “advantage” to mixing the two; that is pure downside, because it makes the game less consistent and forces players to remember which rolls are which.
But picking one and enforcing consistency has its own downside: there can be advantages to one or the other in certain situations, but if you enforce consistency you cannot use the “better” approach when it would make sense. For examples:
Roll-under has the nice property for percentile rolls that your target number is also your chance of success: if you must roll a 20 or less on a d100, you have a 20% chance of succeeding. For roll-over, the same 20% chance would be a roll above 80—you have to do 100−x to determine your odds each time.
On the other hand, roll-over works much better for unbounded numbers: you can always increase a target number in roll-over, but decreasing a target number in roll-under is somewhat awkward when it gets negative.
In short, by not having the game consistently use one or the other, you are free to pick whichever is appropriate for a given roll, not being constrained by the game using the other type for everything else. The downside is, the freer you are with this—the more you use whatever roll type seems most appropriate for each roll, the harder it’s going to be for players to remember which rolls are which.