Using 6/3 NM cable to feed 60 Amp to subpanel

First thing I would do it get a larger sub-panel. Those dinky 6/12 panels are terrible for space and future expansion. You also need a means of disconnect at the outbuilding. I would get a 20 space main breaker panel which would correct both of these issues.

In the sub-panel you would also need to install a grounding bar to the back of the panel box and NOT install the bonding screw. The green wire would terminate there as well as the required grounding electrode (ground rod typically) conductor.

At the house you MUST protect that URD cable where it is exposed. Typically we'd use Sch80 PVC conduit. For the splice you can/should use Polaris connectors. They are available at any electrical supply house or even some home centers. They make larger aluminum wire splices a breeze!

There are a LOT of other code requirements for a job like this. Are you experienced enough to tackle this job?

This one popped up for some reason, so I'll edit in some new thoughts.

Corroded or arcing power feeds are "burn your house down" serious business. Fix immediately! Aluminum feeder wire is fine stuff, but you do need to use the anti-ox goop, and torque it correctly. Many electricians in the 80s and 90s did not get the memo. Nip it in the bud fast.

Likewise corrosion (read: arcing) on breaker contact points is absolutely unacceptable. Catch it early and you may be able to save the breaker space, otherwise that's a dead space in your panel. The #1 cause is using the wrong brand of breaker. Brand X breaker won't engage Brand Y buses with the correct contact shape or clamping force, causing the arcing.

If the panel is a goner, swap it. Let me come back to that.

The cable between main and sub doesn't need 2 breakers. It needs a breaker upstream (in the main panel) to protect the cable. A 60A breaker is correct to protect #6Cu or #4Al. At the subpanel, the cable/wire can land on main lugs (provided the subpanel is rated for 60A+) or on the subpanel's main breaker (of any rating - a smaller main protects the panel, a larger main is just a shutoff switch, which you do not need since it's in the same building.) So if the subpanel has main lugs, use them instead of backfeeding a breaker.

One more thing. If the main panel is breakered at 100A, and the cable supports 100A, and the subpanel supports 100A, then the main main breaker protects them all. You don't need any breakers for the sub. You can use thru-lugs or snap-in lugs in the main panel.

With that in mind, back to replacing the panel(s). Talk to the power company and see if your service drop/meter can support 125A. Even if not, you can order a 125A main panel and swap the main breaker to 100A.

See if you can upgrade the main-sub cable to 125A. If so, order a main panel with thru lugs, which will save 2 breaker spaces, and a 125A sub. Voila, no breakers needed, space maximized!

You can also look at the CH or QO panels, which use 3/4" high breakers, so more spaces in the same panel footprint. GFCI and AFCI breakers are available for them.

What doesn't work is "double-stuff" breakers (including GE's 1/2" Q-line breakers). Those are useless as almost all new work today requires GFCI, AFCI or both - and you cannot obtain those in double-stuff. So a "24/48" panel is actually 24, period.

Original:

Wire gauge is decided by what NEC requires (and those are in stair-steps) and what your electrical distributor has in stock.

So with numbers as close as 100A vs 125A, you might get lucky: imagine the original installer found #2 wire was good for 95A, #1 for 110A and #0 aka 1/0 was good for 125A. He can't use #2, the distributor does not stock #1, so he uses 1/0. You get a happy surprise when you aim to upgrade. Hey, it could happen - check. If not, you need to pull bigger wire.

Check the markings on the aluminum conductors to make sure they are stranded and the modern AA-8000 series alloy, which are legal and safe. (NEC 310.106b). The problem alloys from the 1950s are now outlawed, and your distributor won't even sell them.

Your reduced voltage to 108 is more likely to be a problem with your neutral. Poor connections can't drop 12 volts without destroying themselves from heat (volts x amps = watts of heat), more likely your other leg went to 132 volts because of a lost neutral. This is an extremely dangerous condition that needs to be taken out of service immediately because the imbalance can be much worse than 12V, and that will destroy appliances and start fires. If you know where the problem is, but can't de-energize your house now, a short-term work-around is disable one "hot" - doesn't matter which one. Look at how the hot busbars are arranged inside the panel, pick one leg, and remove every breaker which uses that leg. Rearrange as necessary to keep the lights on. At that point, if the neutral fails, the circuits will shut off instead of giving dangerous voltages.

Look for other options (i.e. other brands or suppliers) that will give you more spaces in the panel. You don't want to be forced into using duplex breakers, because - especially when you remodel - you'll be forced to follow current code, and will need more kitchen breakers and GFCI and AFCI breakers in many slots. Those are a lot more expensive as duplex. You can completely fill the panel with double-stuff breakers (in fact, doing so is the basis of their "48 circuit" marketing claim).

Don't buy this at the big-box home improvement stores, go to a real electrical distributor such as Greybar - they tend to be locally owned. They will have better options (eg more breaker spaces in smaller boxes, Homelites are huge), better quality and sometimes better prices. For instance, Homelite is Square D's bottom-tier brand. There's better out there, and it's worth an extra $100 to not have problems like the ones you are having.