[RPG] Syntax for a unique dice system on Anydice

An exhaustive generation of the possibilities can be done.

It is not 100% clear what you mean by "dice totalling at least 5", I've chosen to interpret this as "sum up to 5 dice" and that means one would have at least one success with a pool of 5 or more dice (and at least 2 successes with a pool of 10 dice).

In principle, the algorithm is simple:

# interpret "roll a die" as "loop from 1 to 6 and for each..."
declare a tabulation function
declare a "generate all rolls" function, taking dice to roll and rolls so far
  if we should roll only one more dice:
    roll it, add it to the rolls so far, pass that to the tabulator
  otherwise:
    roll a die, add it to the rolls so far and call ourselves, generating one less roll

Then you can tabulate all the number of different successes. So far, I have the results for pools of size 2-9 in and can edit the entry with the results.

What you need to do is write a function to "freeze" your hit die rolls, and call it recursively for each (unsuccessful) attack, something like this basic example:

ATK:  1d20  \ attack roll: use 1@2d20 for advantage or 2@2d20 for disadvantage \
HIT:    +0  \ bonus to attack roll \
AC:      8  \ target AC: attack hits if ATK + HIT >= AC \
CRIT:   20  \ minimum attack roll to crit \
DMG:   1d4  \ damage rolled on successful hit \
BONUS: 1d6  \ bonus damage on first hit \

function: attack ROLL:n {
  \ return the number of times damage is applied on an attack roll of ROLL \
  if ROLL <= 1 { result: 0 }     \ natural 1 is always a miss \
  if ROLL >= CRIT { result: 2 }  \ critical hit deals damage damage twice \
  result: ROLL + HIT >= AC
}

function: multiattack N:n times {
  result: [roll [attack ATK] to multiattack N times]
}
function: roll FIRSTHIT:n to multiattack N:n times {
  if N <= 0 {
    result: 0
  }
  if FIRSTHIT > 0 {
    DAMAGE: FIRSTHIT d (DMG + BONUS)
    loop I over {2..N} {
      DAMAGE: DAMAGE + [attack ATK] d DMG
    }
    result: DAMAGE
  } else {
    result: [roll [attack ATK] to multiattack N-1 times]
  }
}

output [multiattack 3 times]

The main [multiattack N:n times] function just calls the helper function [roll FIRSTHIT:n to multiattack N:n times]. The first argument (FIRSTHIT) to the helper function is given by [attack ATK], which returns the number of times damage is to be applied for this attack roll (i.e. 0 for miss, 1 for hit and 2 for crit).

Since this argument is marked as numeric (with :n) in the function definition, AnyDice internally calls the helper function with each possible return value of [attack ATK], and weighs its results according to the probabilities of the outcomes. This means that, even though the result of [attack ATK] is a (biased) die, inside the helper function its value is "frozen" as FIRSTHIT, which is a fixed number and thus can be used e.g. in if conditionals. (See the "Functions" section of the AnyDice documentation for more details.)

The helper function itself just checks if the attack hits (i.e. if FIRSTHIT > 0), and if so, applies the damage from this attack (plus the bonus, if any) and from any remaining attacks (without the bonus). On the other hand, if the attack misses, the helper function instead calls itself recursively with a new attack roll, and with the number of remaining attacks reduced by one.

Ps. The expression FIRSTHIT d (DMG + BONUS) may be worth discussing: it simply rolls the normal and bonus damage FIRSTHIT times, and adds up the results. (On the other hand, FIRSTHIT * (DMG + BONUS) would yield the result of rolling the normal and bonus damage once, and multiplying it by FIRSTHIT.) Thus, if FIRSTHIT equals 2, the damage (including any Sneak Attack or other bonuses) will be rolled twice, as the 5e rules specify for critical hits. Similarly, inside the loop, the expression [attack ATK] d DMG gives the result of rolling DMG either zero, one or two times, according to the probabilities given by the [attack ATK] roll.

In fact, all the code inside the if FIRSTHIT > 0 conditional block could be replaced with the following expression:

result: FIRSTHIT d (DMG + BONUS) + (N-1) d ([attack ATK] d DMG)

Here, (N-1) d ([attack ATK] d DMG) gives the result of performing N-1 attack rolls, where each attack deals DMG damage [attack ATK] times.

Also, it's worth noting that AnyDice normally allows function calls to be nested only 10 levels deep (one of which is used up by the wrapper function in this example, and one by the [attack ATK] call). If you'd like to support multiattacks with more than 8 successive attack rolls, you should increase the recursion limit e.g. like this:

set "maximum function depth" to 999

However, to make such deep recursion run in a reasonable time, it's essential to make sure that the helper function call itself only on one specific value of its non-fixed parameters (e.g. in this case, when FIRSTHIT is zero). If it can recurse on two or more possible inputs, then the number of different possible sequences of recursive calls that AnyDice needs to check will grow exponentially with the recursion depth.

This is the reason why the code above uses the result of [attack ATK] (which can only be 0, 1 or 2) as the parameter for the helper function, instead of passing the attack roll ATK (which could be anything from 1 to 20) directly as a parameter. While doing it that way will work OK for a small number of attacks, it gets very slow very quickly as the recursion depth is increased.

In fact, we could optimize the code a bit more by precalculating the results of [attack ATK] and assigning them to a global custom die. This would speed up the code slightly, and also save one function nesting level.

Also note that the code above assumes that the bonus damage will always be applied on the first successful hit, regardless of whether it's a crit or not. This is probably optimal in most practical situations, but in principle, a player with a sufficiently large number of attacks remaining and a sufficiently high hit and/or crit change might be better off saving their Sneak Attack for later if they roll an early non-critical hit, on the assumption they'll probably roll a crit (or at least another normal hit) later. Modeling such advanced strategies is possible in AnyDice (generally also via recursion), but more complicated.