It looks like that panel has two grounding bars. One on each side, mostly hidden behind insulated wires in your picture. It also appears like the Neutral bus bar is distinct, and care was taken to keep neutrals and grounds on their separate bars. This will save you much time.
By the looks of your picture, the panel chassis is used as part of the ground bus, which is kind of sketchy, but ok. I would also guess the brass screw on the neutral bar doesn't have any hole to put a wire through, and touches an unpainted section of the chassis. if so, removing it will likely separate the ground and neutral plane in that breaker cabinet.
After doing that (and testing with a multimeter that ground and neutral are separated), you should be able to run this panel as a sub off of a 100A breaker in the new panel.
Does the sub-panel seem over loaded? If so, I could keep the water-heater in the main panel and free up space in the panel another way.
Seems reasonable to me. Most of the equipment won't draw anywhere near the overcurrent rating, at least not during normal operation. Motor loads will draw a higher current on start, but you shouldn't have a problem.
I know I need four-strand wire to run to the sub-panel (2 hot, neutral, ground) but copper or aluminum and what gauge?
You can use either copper or aluminum, though I recommend copper for DIYers. Copper is quite a bit more expensive, but it's easier to work with (in my opinion). If you feel confident working with aluminum conductors, you can save some money using it.
I've covered the topic of feeder sizing here, so I won't go into detail. If you're using copper, you'll want to use 3 AWG conductors. If you choose to use aluminum, you'll need 1 AWG conductors.
If you want to run a single cable, instead of individual conductors in conduit. You can buy what's called 3-3-3-5 SER cable (1-1-1-3 for aluminum), which will contain three 3 AWG conductors (hot,hot,neutral) and a 5 AWG grounding conductor.
When I run the wire along the floor joist, does it need to be secured to the joist or can it just hang there and rest on the drop ceiling? Seems like it should be secured to the joist with wire hanger or something.
You'll have to attach the cable to the joists, using 1 - 1 1/4" staples or other approved means. Check the packaging, to make sure they are rated for the size cable you're using.
What are the things about this project that I don't know that I don't know. :) These are the scary things IMO...i.e. the questions I don't know enough to ask.
The cable you'll be working with is thick and heavy, and it's not going to be fun pulling it. You'll probably want a couple helpers, to help you wrangle it.
Make sure all your connections are tightened to the manufacturer's specified torque.
If you choose aluminum conductors, make doubly sure you tighten the connections. And don't forget the anti-oxidant.
Come back a day or two after the panel has been put into service, and tighten any connections that need it.
Don't forget to remove the bonding jumper between the grounded and grounding bus bars.
You'll need clamps big enough for the cable, to secure it to the panels.
should I put a 100 amp breaker in the sub-panel to act as the "main" for the sub-panel? Or is the 100 amp breaker in the main panel sufficient?
You can usually pick up a main breaker panel, for about the same price as a main lug only (MLO) panel. In my opinion, unless the secondary panel is next to; or within sight of, the main panel. You're better served to install a main breaker panel. It simply offers better protection during maintenance, or other work within the panel.
For example. If you turn off the feeder breaker in the main panel, and start working in the secondary panel. Somebody could easily come along, and flip on the feeder breaker. Since you can't keep an eye on the breaker, you can never be sure the panel will be dead. (unless of course you're using a lockout like you should).
If the secondary panel is in a separate building or structure, then you either need a main breaker, a main disconnect, or the ability to disconnect all ungrounded conductors within 6 or less hand moves.
Best Answer
The problem with your no-splicing plans is the cost of copper
While avoiding splices, boxes, and access points sounds like a good idea, the problem with your plan is that copper 3/3/3/5 SER cable is quite expensive stuff at over $5/ft yet only a hair smaller at 0.91" diameter vs 1.08" diameter for 1/1/1/3 Al. This means that you're looking at a few hundred dollars of SER cable alone for either your plan 3 or the plan 4 you mentioned in your comment, and makes the parts required for a splice rather cost-effective in comparison. Going to 1/1/1/3 aluminum SER at about $1.60-$1.70/ft makes avoiding splicing a cost-effective option, though, and in that vein, I would favor plan 4 with cutting the existing feeder and rerouting it, then replacing the rest of the existing feeder run with a new run. (Aluminum cable is also lighter than copper cables with the same ampacity, which factors into the ease of pulling significantly at the larger gauges that aluminum wire is typically used for.)
If you do go that route, you'll want to use a subfeed lug block in the new subpanel to feed the existing panel instead of a 100A breaker since the breaker would be fairly pointless anyway due to the inability to coordinate it with the existing feeder breaker. If you really want a disconnect for the original subpanel in the new subpanel, then you can use a 100A molded case switch of the appropriate type for the new subpanel.
If you are to splice....
If you do go with the splice-box plans, there are a few points you will need to keep in mind. First off, the T-splice is a much better idea (less cable and fewer splices) than the "loop" splice, so we'd be going with your plan 1 in this case instead of your plan 2. Furthermore, you will need to have the splice box be accessible now and forevermore; you can paint or otherwise finish the cover to match the surroundings, but actually burying a box is no good. Finally, we'll have to account for the NEC 300.14 requirement of 6" minimum of free conductor for each cable entering the box; if you can't get that from slack in the existing run, you'll have to go up to a 12" long by 6" to 8" wide by 4" deep pull box and add an extra block and some 2AWG copper jumpers to connect the two, as well as a spare grounding lug such as a Brumall 6T attached to the box with a 10-32 self-tapping/self-drilling grounding screw such as the Garvin GSST.
Getting into the details of making the splice, we'll first need a big box, far larger than the junction boxes you're used to, due to NEC 314.28(A). In particular, you'll need an absolute minimum of a 6" long by 8" wide by 3" or 4" deep, NEMA 1 (indoor) pull box with a flush-mount cover, grounding kit, and 8AWG grounding jumper in order to accommodate such a tap, using dual-rated (AlxCu) insulated mechanical splicing connectors (Polaris™ or equivalent). However, it may be cheaper to use a larger box so that you can use a UL listed (UL1953/QPQS) power distribution block in accordance with NEMA 314.28(E) as Polaris-type connectors tend to be rather expensive compared to distribution blocks, and also lead to a somewhat messier install compared to the neatness a distribution block can provide in this application.
If you do decide to go the distribution block route, you'll need an 8" long by 8" wide by 4" deep NEMA 1 pull box with a flush cover, a dual-rated (AlxCu) 3-pole power distribution block such as the MPDB63193 to make the splices, a few appropriately sized sheet-metal screws to mount the PDB to the box, and some grounding parts. In your case, I would use an Ilsco NBAE-0307-1 mounted with a 1" 10-32 machine screw through the supplied grounding hole in the box to attach the ground wires to the box. If the box lacks a grounding hole, you can use a 1" 10-32 thread-rolling screw (Taptite™ or equivalent) into an appropriate field-drilled pilot hole instead of the 10-32 machine screw.
Either way, you'll need appropriate bushings and SER clamps to get the cables into the box. You may also need to make your own KOs in the sides of the pull box using a punch set in order to fit the SER clamps in the appropriate places so that you have sufficient slack in the existing wires to make your connections. Some pull boxes ship without knockout rings pre-stamped in them to begin with, even!
Go big or go home!
If you don't already have the new panel installed, I would get the biggest panel I could in order to save you the hassle of having to upgrade it down the line. In particular, a 24- or 30-space, 125A, main lug panel would not be at all out of place here. If you feel like putting the money you're saving on wire to work, you can even use a 40-space or 42-space, 200A or 225A main lug panel for the new subpanel. Either way, you'll need to fit the panel with separate grounding bars and pull the bonding screw from it, just like any other subpanel.
TORQUE ALL LUGS TO SPEC
Last but not least, you'll want to use an inch-pound torque wrench to torque all the various connection setscrews to their labeled torque specifications. This is a new requirement in NEC 110.14(D) as of the 2017 NEC, and a good idea anyway in order to ensure that your connections will pass the test of time.