[RPG] How to implement this specialized roll-and-keep mechanic in AnyDice

Here's a straightforward brute-force solution:

function: compare A:s vs B:s {
  SA: A >= 1@B
  SB: B >= 1@A
  result: (SA > SB) - (SA < SB)
}
output [compare 5d6 vs 4d6]

It just iterates over all the possible rolls for each player (this happens automatically in AnyDice when you pass dice to a function expecting sequences), calculates the number of successes for each player, and returns -1, 0 or +1 depending on whether A got less then, as many as or more successes than B.

Implementing the tie-breaker turns out to be even simpler:

function: compare A:s vs B:s {
  result: (A > B) - (A < B)
}
output [compare 5d6 vs 4d6]

The reason this works is that, when you compare two sequences in AnyDice, they're compared lexicographically from left to right (i.e. from highest to lowest, in the default sort order). Thus, e.g. {6,5,5,1} > {6,5,4,4} is true (i.e. returns 1), because the first two elements of both sequences are equal, but the third element is greater in the first sequence.

(A bit confusingly, comparing a sequence to a number does something different: it compares the number to each element in the sequence, and returns the number of elements for which the comparison is true. And comparing a sequence to a die does something different again. It doesn't help that the documentation is kind of ambiguously phrased, and could be interpreted to mean that comparing two sequences would return the number of successful comparisons between respective elements. A quick test will show that this is not the case, however.)

As it happens, this is precisely equivalent to your tie-breaking rule, at least as far as determining the winner goes. To show why, let me first write your rules in a more compact (but hopefully obviously equivalent) form:

1. If one player's highest die is greater than the other's (or if only one player has any dice left and the other doesn't), that player wins.
2. If the highest dice are tied, the player with more of those dice wins.
3. (tie-breaker) If the players have the same number of highest dice, those dice are discarded, and we repeat from step 1. (If both players simultaneously run out of dice, the roll is a perfect tie. This can only happen if both players have the same number of dice to begin with.)

Now, it should not be hard to see that steps 2 and 3 above can equivalently be replaced with the following alternative rule:

2. (alternative) If the highest dice are tied, discard (only!) those two dice, and repeat from step 1.

Clearly, if one player would win under the original rule 2, they will also win under (sufficiently many iterations) of the alternative rule 2. Similarly, if the players' highest dice are tied, and get discarded under the original rule 3, it should be clear that applying the alternative rule sufficiently many times will accomplish the same result.

The accepted solution disregards the core purpose of the actual question (how does user choice affect the outcome), so I had to try and figure it out...

I'm new to anydice, so I welcome corrections/suggestions. Here, if the player rolls their desired outcome on 2/3 dice, that becomes their choice, otherwise, output a random choice from the 3....

function: choice of A:n and B:n and C:n target T:n {
  if A=T { result: A }
  if B=T { result: B }
  if C=T { result: C }
  result: 1d{A, B, C}
}
function: combo of A:n and B:n and C:n target T {
  result: [choice of A+B and B+C and A+C target T]
}

loop T over {1..12} {
  output [combo of 1d6 and 1d6 and 1d6 target T] named "2/3d6 [T]"
}

Here, we can see the huge difference that player choice makes

Chance of getting a 7:

2d6: ~17%

(2/3)d6: ~42%

This matches the mathematical answer:

The probability that a pair of the three dice has sum '7' is $$\frac{3 \cdot 6 \cdot 4 + 3 \cdot 6}{6^3} = \frac{90}{216} = \frac{5}{12} ≈ 42/100 $$ Src: https://math.stackexchange.com/questions/3283971/probability-that-2-dice-selected-from-3-rolled-dice-will-have-sum-7