[RPG] Help Writing an AnyDice Function for a Weird Dice Mechanic

An exhaustive generation of the possibilities can be done.

It is not 100% clear what you mean by "dice totalling at least 5", I've chosen to interpret this as "sum up to 5 dice" and that means one would have at least one success with a pool of 5 or more dice (and at least 2 successes with a pool of 10 dice).

In principle, the algorithm is simple:

# interpret "roll a die" as "loop from 1 to 6 and for each..."
declare a tabulation function
declare a "generate all rolls" function, taking dice to roll and rolls so far
  if we should roll only one more dice:
    roll it, add it to the rolls so far, pass that to the tabulator
  otherwise:
    roll a die, add it to the rolls so far and call ourselves, generating one less roll

Then you can tabulate all the number of different successes. So far, I have the results for pools of size 2-9 in and can edit the entry with the results.

Here's a straightforward brute-force solution:

function: compare A:s vs B:s {
  SA: A >= 1@B
  SB: B >= 1@A
  result: (SA > SB) - (SA < SB)
}
output [compare 5d6 vs 4d6]

It just iterates over all the possible rolls for each player (this happens automatically in AnyDice when you pass dice to a function expecting sequences), calculates the number of successes for each player, and returns -1, 0 or +1 depending on whether A got less then, as many as or more successes than B.

Implementing the tie-breaker turns out to be even simpler:

function: compare A:s vs B:s {
  result: (A > B) - (A < B)
}
output [compare 5d6 vs 4d6]

The reason this works is that, when you compare two sequences in AnyDice, they're compared lexicographically from left to right (i.e. from highest to lowest, in the default sort order). Thus, e.g. {6,5,5,1} > {6,5,4,4} is true (i.e. returns 1), because the first two elements of both sequences are equal, but the third element is greater in the first sequence.

(A bit confusingly, comparing a sequence to a number does something different: it compares the number to each element in the sequence, and returns the number of elements for which the comparison is true. And comparing a sequence to a die does something different again. It doesn't help that the documentation is kind of ambiguously phrased, and could be interpreted to mean that comparing two sequences would return the number of successful comparisons between respective elements. A quick test will show that this is not the case, however.)

As it happens, this is precisely equivalent to your tie-breaking rule, at least as far as determining the winner goes. To show why, let me first write your rules in a more compact (but hopefully obviously equivalent) form:

1. If one player's highest die is greater than the other's (or if only one player has any dice left and the other doesn't), that player wins.
2. If the highest dice are tied, the player with more of those dice wins.
3. (tie-breaker) If the players have the same number of highest dice, those dice are discarded, and we repeat from step 1. (If both players simultaneously run out of dice, the roll is a perfect tie. This can only happen if both players have the same number of dice to begin with.)

Now, it should not be hard to see that steps 2 and 3 above can equivalently be replaced with the following alternative rule:

2. (alternative) If the highest dice are tied, discard (only!) those two dice, and repeat from step 1.

Clearly, if one player would win under the original rule 2, they will also win under (sufficiently many iterations) of the alternative rule 2. Similarly, if the players' highest dice are tied, and get discarded under the original rule 3, it should be clear that applying the alternative rule sufficiently many times will accomplish the same result.