Ironsworn meets BitD dice mechanic (with a twist)

An exhaustive generation of the possibilities can be done.

It is not 100% clear what you mean by "dice totalling at least 5", I've chosen to interpret this as "sum up to 5 dice" and that means one would have at least one success with a pool of 5 or more dice (and at least 2 successes with a pool of 10 dice).

In principle, the algorithm is simple:

# interpret "roll a die" as "loop from 1 to 6 and for each..."
declare a tabulation function
declare a "generate all rolls" function, taking dice to roll and rolls so far
  if we should roll only one more dice:
    roll it, add it to the rolls so far, pass that to the tabulator
  otherwise:
    roll a die, add it to the rolls so far and call ourselves, generating one less roll

Then you can tabulate all the number of different successes. So far, I have the results for pools of size 2-9 in and can edit the entry with the results.

Efficient computation

The trouble with AnyDice is that it works by enumerating all possible multisets of numbers rolled. While this is faster than enumerating all ordered (equivalently, unsorted) rolls and is convenient in that it gives you the entire rolled sequence at once, the number of possible multisets can still grow quickly as you add dice. Mixing different dice also makes things much more difficult.

Fortunately, it turns out that, as long as you can express the dice mechanic in terms of an incremental calculation where you're told how many dice rolled each number in each pool for one number at a time without carrying too much information from number to number, it's possible to compute the solution more efficiently. It's also possible to handle mixed standard dice without much loss in efficiency. I've implemented this approach in my Icepool Python package. Here's a script:

import icepool
from icepool import d4, d6, d8, d10, d12, EvalPool

class GreenRed(EvalPool):
    def next_state(self, state, outcome, green, red):
        # State is the number of top-two dice for green and red.
        top_green, top_red = state or (0, 0)
        # If there are remaining places in the top two...
        remaining_top_two = 2 - (top_green + top_red)
        if remaining_top_two > 0:
            # Compute the number of non-eliminated dice that rolled this outcome.
            net = green - red
            # Then add them to the winning team's top two.
            if net > 0:
                top_green += min(net, remaining_top_two)
            elif net < 0:
                top_red += min(-net, remaining_top_two)
        return top_green, top_red
    
    def final_outcome(self, final_state, *pools):
        top_green, top_red = final_state
        if (top_green > 0) and not (top_red > 0): return 2
        elif (top_red > 0) and not (top_green > 0): return 0
        else: return 1
    
    def direction(self, *_):
        # See outcomes in descending order.
        return -1
    
green_red = GreenRed()
# The argument lists are implicitly cast to pools.
print(green_red.eval([d10, d8], [d6, d8]))

Denominator: 3840

We can handle even much larger pools. Here's two entire dice sets (10 dice total) on each side:

print(green_red.eval([d12, d10, d8, d6, d4, d12, d10, d8, d6, d4], [d12, d10, d8, d6, d4, d12, d10, d8, d6, d4]))

Denominator: 281792804290560000

You can try this in your browser using this JupyterLite notebook. Fair warning, though, I'm currently doing a major revision to the package.