Have you considered a remote for the house end of the garage hookup? I'm thinking a z-wave controllable light switch or something in the garage, and a remote control in the house. That way you can ignore the 12/3 romex and weird wiring (of that part of the circuit) and get on with your day.
I'm unclear whether it's allowed by code (especially with the colors you have), but with 3 wires, you've got enough for a single 30 amp circuit - ground, neutral and hot. Is that enough for the garage? That would at least be safe.
Have you tried searching the site? The panel feeder size question has been answered many times. Here's one such example.
Is you're planning to pull a cable, you'll use 3-3-3-5 USE cable with copper conductors (1-1-1-3 for aluminum). If you're pulling individual wires, you'll want four 3 AWG THWN wires (hot, hot, neutral, ground)(black, red, white, green) (1 AWG for aluminum). If the run is going to be really long, you'll want to calculate the voltage drop and adjust the conductor size accordingly.
If you use the 3-3-3-5 USE cable, you don't need conduit. If you want to install it in conduit, you'll need 1 1/4" schedule 40 PVC. If you use individual THWN wires, you'll also use 1 1/4" schedule 40 PVC.
You'll have to keep the grounded (neutral), and grounding bars separate in the secondary panel.
You'll have to install a grounding electrode system at the garage, and bond the equipment and grounding conductor to it.
NOTES: This is a very broad, and generalized overview, which lacks specific details for your particular situation. You may need the advice of a licensed Electrician, to provide a more detailed plan based on the actual job site.
Best Answer
I would go for the tone generator, because I already have one anyway and use it for business, so it's paid for. But if I didn't have one and did have a multimeter (everyone should have a multimeter), then I would actually test as follows, knowing that there are 12/3 cables rather than random individual wires:
If you have some resistors around of known values, you could do everything in one batch, but with only 3 cables, this is about as easy as it gets. Actually, thank you @cube for a way to do essentially that, by treating different combinations of wires in each cable differently, which allows you to differentiate 3 cables at one time without any resistors. This could actually be extended up to decoding as many as 5 cables at a time:
Extending to more than 5 would actually become 4-at-a-time because you wouldn't know the difference between any of the "None" cables.