Does the sub-panel seem over loaded? If so, I could keep the water-heater in the main panel and free up space in the panel another way.
Seems reasonable to me. Most of the equipment won't draw anywhere near the overcurrent rating, at least not during normal operation. Motor loads will draw a higher current on start, but you shouldn't have a problem.
I know I need four-strand wire to run to the sub-panel (2 hot, neutral, ground) but copper or aluminum and what gauge?
You can use either copper or aluminum, though I recommend copper for DIYers. Copper is quite a bit more expensive, but it's easier to work with (in my opinion). If you feel confident working with aluminum conductors, you can save some money using it.
I've covered the topic of feeder sizing here, so I won't go into detail. If you're using copper, you'll want to use 3 AWG conductors. If you choose to use aluminum, you'll need 1 AWG conductors.
If you want to run a single cable, instead of individual conductors in conduit. You can buy what's called 3-3-3-5 SER cable (1-1-1-3 for aluminum), which will contain three 3 AWG conductors (hot,hot,neutral) and a 5 AWG grounding conductor.
When I run the wire along the floor joist, does it need to be secured to the joist or can it just hang there and rest on the drop ceiling? Seems like it should be secured to the joist with wire hanger or something.
You'll have to attach the cable to the joists, using 1 - 1 1/4" staples or other approved means. Check the packaging, to make sure they are rated for the size cable you're using.
What are the things about this project that I don't know that I don't know. :) These are the scary things IMO...i.e. the questions I don't know enough to ask.
The cable you'll be working with is thick and heavy, and it's not going to be fun pulling it. You'll probably want a couple helpers, to help you wrangle it.
Make sure all your connections are tightened to the manufacturer's specified torque.
If you choose aluminum conductors, make doubly sure you tighten the connections. And don't forget the anti-oxidant.
Come back a day or two after the panel has been put into service, and tighten any connections that need it.
Don't forget to remove the bonding jumper between the grounded and grounding bus bars.
You'll need clamps big enough for the cable, to secure it to the panels.
should I put a 100 amp breaker in the sub-panel to act as the "main" for the sub-panel? Or is the 100 amp breaker in the main panel sufficient?
You can usually pick up a main breaker panel, for about the same price as a main lug only (MLO) panel. In my opinion, unless the secondary panel is next to; or within sight of, the main panel. You're better served to install a main breaker panel. It simply offers better protection during maintenance, or other work within the panel.
For example. If you turn off the feeder breaker in the main panel, and start working in the secondary panel. Somebody could easily come along, and flip on the feeder breaker. Since you can't keep an eye on the breaker, you can never be sure the panel will be dead. (unless of course you're using a lockout like you should).
If the secondary panel is in a separate building or structure, then you either need a main breaker, a main disconnect, or the ability to disconnect all ungrounded conductors within 6 or less hand moves.
I would start by putting a rather large subpanel as close as possible to it, probably whichever Siemens panel ThreePhaseEel recommends, or a CH or QO if space is an issue. 42+ space. Position it with an eye toward rerouting the service wiring there.
Even before that, talk to the power company about upgrade options. If a particular size is possible, get the subpanel with that main breaker, but don't sacrifice spaces to do it.
Short term, if the new subpanel is >=100A it doesn't need any main breaker at all, or even a main-panel breaker. The main breaker in the main panel will protect it. Use at least 100A wire to connect the panels, i.e. At least #3Cu or #1Al. A reason to use cheaper Al on such a short run is the lugs will be aluminum.
Having a breaker in the main panel can be avoided with a lug-breaker, which snaps in like a breaker but has only lugs.
Now, at leisure, move one circuit at a time from the old panel to the new. That's just to get away from those somewhat dodgy Challenger breakers. If wires don't reach, I typically put 4-11/16" square boxes at opportune locations to splice. Ideally metal conduit the rest of the way to the new panel, and THHN wire inside the conduit. Ideally.
With all the circuits moved over, try to straightline the subpanel straight off the bottom (switched side) of the main breaker.
Alternately, if the power company's upgrade path calls for a new meter pan, get one with a main breaker. Gut the old panel, using it only as a gutter, to the new subpanel, using wires rated for main breaker current. If that's the plan, you can use a main lug subpanel instead of a main breaker one, saving some space. 42-space main lug CHs are not that big.
Best Answer
It looks like your old Westinghouse panel will accept breakers from Eaton or Cutler-Hammer (but check at place of purchase). If so, this tandem breaker can serve two circuits but use only one slot in the panel.
This is different from a double pole breaker which takes two slots like you would need for a 240 volt appliance:
To use the tandem, pull out a pair of single pole/non-tandem breakers (and discard), connect the two circuit wires to the tandem and insert. Voila! You have freed up an extra slot.