Does the sub-panel seem over loaded? If so, I could keep the water-heater in the main panel and free up space in the panel another way.
Seems reasonable to me. Most of the equipment won't draw anywhere near the overcurrent rating, at least not during normal operation. Motor loads will draw a higher current on start, but you shouldn't have a problem.
I know I need four-strand wire to run to the sub-panel (2 hot, neutral, ground) but copper or aluminum and what gauge?
You can use either copper or aluminum, though I recommend copper for DIYers. Copper is quite a bit more expensive, but it's easier to work with (in my opinion). If you feel confident working with aluminum conductors, you can save some money using it.
I've covered the topic of feeder sizing here, so I won't go into detail. If you're using copper, you'll want to use 3 AWG conductors. If you choose to use aluminum, you'll need 1 AWG conductors.
If you want to run a single cable, instead of individual conductors in conduit. You can buy what's called 3-3-3-5 SER cable (1-1-1-3 for aluminum), which will contain three 3 AWG conductors (hot,hot,neutral) and a 5 AWG grounding conductor.
When I run the wire along the floor joist, does it need to be secured to the joist or can it just hang there and rest on the drop ceiling? Seems like it should be secured to the joist with wire hanger or something.
You'll have to attach the cable to the joists, using 1 - 1 1/4" staples or other approved means. Check the packaging, to make sure they are rated for the size cable you're using.
What are the things about this project that I don't know that I don't know. :) These are the scary things IMO...i.e. the questions I don't know enough to ask.
The cable you'll be working with is thick and heavy, and it's not going to be fun pulling it. You'll probably want a couple helpers, to help you wrangle it.
Make sure all your connections are tightened to the manufacturer's specified torque.
If you choose aluminum conductors, make doubly sure you tighten the connections. And don't forget the anti-oxidant.
Come back a day or two after the panel has been put into service, and tighten any connections that need it.
Don't forget to remove the bonding jumper between the grounded and grounding bus bars.
You'll need clamps big enough for the cable, to secure it to the panels.
should I put a 100 amp breaker in the sub-panel to act as the "main" for the sub-panel? Or is the 100 amp breaker in the main panel sufficient?
You can usually pick up a main breaker panel, for about the same price as a main lug only (MLO) panel. In my opinion, unless the secondary panel is next to; or within sight of, the main panel. You're better served to install a main breaker panel. It simply offers better protection during maintenance, or other work within the panel.
For example. If you turn off the feeder breaker in the main panel, and start working in the secondary panel. Somebody could easily come along, and flip on the feeder breaker. Since you can't keep an eye on the breaker, you can never be sure the panel will be dead. (unless of course you're using a lockout like you should).
If the secondary panel is in a separate building or structure, then you either need a main breaker, a main disconnect, or the ability to disconnect all ungrounded conductors within 6 or less hand moves.
This one popped up for some reason, so I'll edit in some new thoughts.
Corroded or arcing power feeds are "burn your house down" serious business. Fix immediately! Aluminum feeder wire is fine stuff, but you do need to use the anti-ox goop, and torque it correctly. Many electricians in the 80s and 90s did not get the memo. Nip it in the bud fast.
Likewise corrosion (read: arcing) on breaker contact points is absolutely unacceptable. Catch it early and you may be able to save the breaker space, otherwise that's a dead space in your panel. The #1 cause is using the wrong brand of breaker. Brand X breaker won't engage Brand Y buses with the correct contact shape or clamping force, causing the arcing.
If the panel is a goner, swap it. Let me come back to that.
The cable between main and sub doesn't need 2 breakers. It needs a breaker upstream (in the main panel) to protect the cable. A 60A breaker is correct to protect #6Cu or #4Al. At the subpanel, the cable/wire can land on main lugs (provided the subpanel is rated for 60A+) or on the subpanel's main breaker (of any rating - a smaller main protects the panel, a larger main is just a shutoff switch, which you do not need since it's in the same building.) So if the subpanel has main lugs, use them instead of backfeeding a breaker.
One more thing. If the main panel is breakered at 100A, and the cable supports 100A, and the subpanel supports 100A, then the main main breaker protects them all. You don't need any breakers for the sub. You can use thru-lugs or snap-in lugs in the main panel.
With that in mind, back to replacing the panel(s). Talk to the power company and see if your service drop/meter can support 125A. Even if not, you can order a 125A main panel and swap the main breaker to 100A.
See if you can upgrade the main-sub cable to 125A. If so, order a main panel with thru lugs, which will save 2 breaker spaces, and a 125A sub. Voila, no breakers needed, space maximized!
You can also look at the CH or QO panels, which use 3/4" high breakers, so more spaces in the same panel footprint. GFCI and AFCI breakers are available for them.
What doesn't work is "double-stuff" breakers (including GE's 1/2" Q-line breakers). Those are useless as almost all new work today requires GFCI, AFCI or both - and you cannot obtain those in double-stuff. So a "24/48" panel is actually 24, period.
Original:
Wire gauge is decided by what NEC requires (and those are in stair-steps) and what your electrical distributor has in stock.
So with numbers as close as 100A vs 125A, you might get lucky: imagine the original installer found #2 wire was good for 95A, #1 for 110A and #0 aka 1/0 was good for 125A. He can't use #2, the distributor does not stock #1, so he uses 1/0. You get a happy surprise when you aim to upgrade. Hey, it could happen - check. If not, you need to pull bigger wire.
Check the markings on the aluminum conductors to make sure they are stranded and the modern AA-8000 series alloy, which are legal and safe. (NEC 310.106b). The problem alloys from the 1950s are now outlawed, and your distributor won't even sell them.
Your reduced voltage to 108 is more likely to be a problem with your neutral. Poor connections can't drop 12 volts without destroying themselves from heat (volts x amps = watts of heat), more likely your other leg went to 132 volts because of a lost neutral. This is an extremely dangerous condition that needs to be taken out of service immediately because the imbalance can be much worse than 12V, and that will destroy appliances and start fires. If you know where the problem is, but can't de-energize your house now, a short-term work-around is disable one "hot" - doesn't matter which one. Look at how the hot busbars are arranged inside the panel, pick one leg, and remove every breaker which uses that leg. Rearrange as necessary to keep the lights on. At that point, if the neutral fails, the circuits will shut off instead of giving dangerous voltages.
Look for other options (i.e. other brands or suppliers) that will give you more spaces in the panel. You don't want to be forced into using duplex breakers, because - especially when you remodel - you'll be forced to follow current code, and will need more kitchen breakers and GFCI and AFCI breakers in many slots. Those are a lot more expensive as duplex. You can completely fill the panel with double-stuff breakers (in fact, doing so is the basis of their "48 circuit" marketing claim).
Don't buy this at the big-box home improvement stores, go to a real electrical distributor such as Greybar - they tend to be locally owned. They will have better options (eg more breaker spaces in smaller boxes, Homelites are huge), better quality and sometimes better prices. For instance, Homelite is Square D's bottom-tier brand. There's better out there, and it's worth an extra $100 to not have problems like the ones you are having.
Best Answer
No, you definitely can't do that with this meter-main! You have several choices.
Put only one panel on the ATS
This is the normal approach to this problem. You pick one panel and make it the ATS-operated panel. Any critical loads in the other panel, you reroute them.
You can also put the other panel on a MTS (manually operated interlock) and those are $100-ish for most panels.
Two ATS's, one for each panel
This is also allowed though at higher cost and complexity, of course.
An ATS that stacks under the electric meter.
Some ATS's have a funny plate that fits between the meter pan and the meter, thus switching the entire service. That type of ATS will work very well in this instance.
Replace the meter-main with something appropriate
Into the trash goes both the meter and the dual main breaker apparatus. Then you fit a meter-main designed to support an ATS, or a plain meter feeding the ATS which then feeds a 200A panel which has the two 100A breakers in it.