How do Challenge Winners Earn More Coins

Here's an AnyDice program to simulate it:

function: test N:n against X:n reroll Y:n later Z:n {
    if N >= Y { result: 1 + [test d6 against X reroll Z later Z] }
    if N >= X { result: 1 }
    result: 0
}
output [test d6 against 4 reroll 6 later 6] named "success on 4-6, reroll on 6"
output [test d6 against 3 reroll 6 later 6] named "success on 3-6, reroll on 6"
output [test d6 against 4 reroll 5 later 6] named "success on 4-6, reroll on 5-6 (re-reroll on 6)"

Looking at the summary output, we can see that the baseline roll (success on 4+, explode on 6) yields on average 0.6 successes per die. Lowering the success threshold to 3+ increases this to an average of 0.8 successes per die, whereas lowering the explode threshold to 5+ gives 0.7 successes per die. (I was a bit surprised that the mean success rates would work out to such nice round numbers, but it seems they do.)

Thus, lowering the success threshold is a significantly better investment than lowering the explosion threshold.

In fact, this is easy to see intuitively: lowering the success threshold by one gives you a 1/6 chance of an extra success; lowering the explosion threshold by one gives you a 1/6 chance of an extra die roll. Given that the expected number of successes per die is less than one, it's pretty obvious that, given the choice, you should choose an extra success over an extra die.

Ps. I did some more testing, and it sems that, if you modify the second ability to also allow successive rerolls on 5+, the average number of successes per die increases to 0.75. That's still less than for the first ability, but closer.

Allowing a reroll on any initial success (i.e. on an initial roll of 4+), but only on a 6 for subsequent rolls, turns out to give the same average number of successes, 0.8, as lowering the success threshold to 3. Thus, if you really want the two abilities to be equally good, that might be a good option to consider.

(Or you could keep the improved reroll ability as originally suggested, i.e. only improving the first reroll chance by 1/6, and make it cost half as much as the improved success ability; if you let the player take the improved reroll ability twice, that would be, on average, equivalent to the first ability.)

The accepted solution disregards the core purpose of the actual question (how does user choice affect the outcome), so I had to try and figure it out...

I'm new to anydice, so I welcome corrections/suggestions. Here, if the player rolls their desired outcome on 2/3 dice, that becomes their choice, otherwise, output a random choice from the 3....

function: choice of A:n and B:n and C:n target T:n {
  if A=T { result: A }
  if B=T { result: B }
  if C=T { result: C }
  result: 1d{A, B, C}
}
function: combo of A:n and B:n and C:n target T {
  result: [choice of A+B and B+C and A+C target T]
}

loop T over {1..12} {
  output [combo of 1d6 and 1d6 and 1d6 target T] named "2/3d6 [T]"
}

Here, we can see the huge difference that player choice makes

Chance of getting a 7:

2d6: ~17%

(2/3)d6: ~42%

This matches the mathematical answer:

The probability that a pair of the three dice has sum '7' is $$\frac{3 \cdot 6 \cdot 4 + 3 \cdot 6}{6^3} = \frac{90}{216} = \frac{5}{12} ≈ 42/100 $$ Src: https://math.stackexchange.com/questions/3283971/probability-that-2-dice-selected-from-3-rolled-dice-will-have-sum-7