Anydice: Reroll the lowest of 3d10 and keep the middle one

Take into account that when you pass 2d10 to Anydice, it automatically sums it. Trying to simulate what you wanted the easiest way I found, here's the code:

function: highestsum A:n B:n C:n {
 result: {1,2} @ [sort {A, B, C}]
}

function: lowestsum A:n B:n C:n {
 result: {2,3} @ [sort {A, B, C}]
}

output [highestsum 1d10 1d10 1d8]
output [lowestsum 1d10 1d10 1d8]

You can test it here.

What both functions do, is sum the three dices and then take out the highest or the lowest. Take into account that it's important to use three dices in this code, splitting the two 2d10 in 1d10 and 1d10.

Edit: Improved thanks to Ilmari Karonen

Doppelgreener's answer is good if the player must always reroll their lowest die, no matter what they originally rolled. However, if using the ability is optional, the player will most likely choose not to use it if they roll, say, two twelves on their first 2d12 roll.

In general, it's hard to model such optional decision-making processes mathematically, since the rationally optimal decision may depend on what the player's specific goal is (not to mention that players are human, and thus often don't act rationally!). However, in this case, a fairly reasonable class of decision-making rules to consider are those where the player rerolls the lowest die only when it's less than some fixed threshold. In fact, if the player's goal is simply to maximize the expected average result of their roll, their optimal strategy is to reroll a die only when the original value of that die is less than the expected average of the reroll (which, for a d12, is (1+12)/2 = 6.5).

Here's a basic AnyDice script to model that decision-making strategy:

function: reroll lowest of ROLL:s as REROLL:d if less than MIN:n {
  LOWEST: (#ROLL)@ROLL       \ the lowest die is sorted last \
  if LOWEST >= MIN {
    result: ROLL + 0
  } else {
    REST: {1..#ROLL-1}@ROLL  \ all but the lowest die \
    result: REST + REROLL
  }
}

output [reroll lowest of 2d12 as d12 if less than 7] named "2d12 replace lowest if < 7"

Note that the function in the code above is generic enough to allow arbitrary initial dice pool sizes (although only the lowest die is ever rerolled) and thresholds, and even provides the option for rerolling with a different die than the original pool had, should that be desired.

Looking at the output of the script, we can see that this "reroll lowest if less than 7" strategy significantly outperforms both "always reroll" and "never reroll":

Of course, we could also consider thresholds other than 7 (≈ 6.5). However, the summary statistics do reveal that, at least as far as the expected average outcome is concerned, 7 is indeed the optimal threshold for rerolling a d12.

All that said, other decision-making rules can still do even better in specific circumstances. For example, if the player is trying to roll to meet or exceed a particular target number, the natural and likely optimal rule is simply to reroll if the sum of their original roll is less than the target, and let the original roll stand otherwise.

Of course, we can model that in AnyDice as well:

function: roll ROLL:s vs TARGET:n with optional REROLL:d reroll {
  SUM: ROLL + 0  \ force the sequence to be summed into a single number! \
  if SUM >= TARGET {
    \ no need to reroll, since we've already met the target \
    result: 1
  } else {
    \ discard and reroll the lowest die \
    REST: {1..#ROLL-1}@ROLL
    result: REST + REROLL >= TARGET
  }
}

loop TARGET over {2 .. 24} {
  output 2d12 >= TARGET named "2d12 vs [TARGET]"
  output [roll 2d12 vs TARGET with optional d12 reroll] named "2d12 vs [TARGET] with optional reroll"
}

In this case, we unfortunately don't get such nice graphs out, since each comparison against the target number just outputs 0 if the roll fails and 1 if it succeeds. Still, looking at the "transposed" view (which I've linked directly to above), we can see that allowing the reroll is slightly better than granting the player +3 to their roll. For example, an unmodified 2d12 roll has a 61.81% chance of meeting a target of 12, whereas 2d12 with an optional reroll has a 64.53% chance of meeting a target of 15, and a 56.71% chance of meeting a target of 16.