Does the sub-panel seem over loaded? If so, I could keep the water-heater in the main panel and free up space in the panel another way.
Seems reasonable to me. Most of the equipment won't draw anywhere near the overcurrent rating, at least not during normal operation. Motor loads will draw a higher current on start, but you shouldn't have a problem.
I know I need four-strand wire to run to the sub-panel (2 hot, neutral, ground) but copper or aluminum and what gauge?
You can use either copper or aluminum, though I recommend copper for DIYers. Copper is quite a bit more expensive, but it's easier to work with (in my opinion). If you feel confident working with aluminum conductors, you can save some money using it.
I've covered the topic of feeder sizing here, so I won't go into detail. If you're using copper, you'll want to use 3 AWG conductors. If you choose to use aluminum, you'll need 1 AWG conductors.
If you want to run a single cable, instead of individual conductors in conduit. You can buy what's called 3-3-3-5 SER cable (1-1-1-3 for aluminum), which will contain three 3 AWG conductors (hot,hot,neutral) and a 5 AWG grounding conductor.
When I run the wire along the floor joist, does it need to be secured to the joist or can it just hang there and rest on the drop ceiling? Seems like it should be secured to the joist with wire hanger or something.
You'll have to attach the cable to the joists, using 1 - 1 1/4" staples or other approved means. Check the packaging, to make sure they are rated for the size cable you're using.
What are the things about this project that I don't know that I don't know. :) These are the scary things IMO...i.e. the questions I don't know enough to ask.
The cable you'll be working with is thick and heavy, and it's not going to be fun pulling it. You'll probably want a couple helpers, to help you wrangle it.
Make sure all your connections are tightened to the manufacturer's specified torque.
If you choose aluminum conductors, make doubly sure you tighten the connections. And don't forget the anti-oxidant.
Come back a day or two after the panel has been put into service, and tighten any connections that need it.
Don't forget to remove the bonding jumper between the grounded and grounding bus bars.
You'll need clamps big enough for the cable, to secure it to the panels.
should I put a 100 amp breaker in the sub-panel to act as the "main" for the sub-panel? Or is the 100 amp breaker in the main panel sufficient?
You can usually pick up a main breaker panel, for about the same price as a main lug only (MLO) panel. In my opinion, unless the secondary panel is next to; or within sight of, the main panel. You're better served to install a main breaker panel. It simply offers better protection during maintenance, or other work within the panel.
For example. If you turn off the feeder breaker in the main panel, and start working in the secondary panel. Somebody could easily come along, and flip on the feeder breaker. Since you can't keep an eye on the breaker, you can never be sure the panel will be dead. (unless of course you're using a lockout like you should).
If the secondary panel is in a separate building or structure, then you either need a main breaker, a main disconnect, or the ability to disconnect all ungrounded conductors within 6 or less hand moves.
Best Answer
200A in the subpanel
Which means you don't actually need a breaker; the existing 200A main breaker will suffice (assuming there actually is one, and you don't have a split-bus/Rule of Six panel).
All you need is subfeed lugs, either built into the panel, or as a bolt-in. (however, this bolt-in will take 4 breaker spaces and limit the use of breaker spaces opposite it; so prepare to move as many as 8 circuits to the new subpanel).
However, this isn't going to work...
These days, people install 200A service because the electric company won't serve anything less, so an all-gas house like mine might not even have 40A on its load calculation. Such a house could support 160A of electric tankless.
However, in the past, people installed 200A service because they had so many electric major appliances that 100A service wasn't enough. Obviously, they do not have 160A of capacity to spare... and I bet you're one of those. If you have 240V breakers on your range, dryer, furnace or A/C, I guarantee your panel does not have 160A of spare ampacity, and you'll need to either change your plan or go with a 400A-class service.
400A-class service actually implements as dual 200A service panels. So your existing 200A panel just stays put. Then you install a second 200A panel (with a LOT of extra spaces this time, please!!) And there you'll have plenty of both physical space and ampacity for your water heaters.
Is this even a good idea?
First, I think the idea of electric water heat is great, if you have the ampacity available to you. Some people don't like it, but I think most of their dislike actually relates to what I'm about to describe.
This isn't a problem you have, certainly... but many people have very bad experiences by getting too small a tankless. The heater could not carry their load (and they didn't do too much thinking about GPM and things like that). These same people also have trouble with electric cars, off-grid solar tech, and other technologies.
Then of course there's the problem you do have, which is that a successfully sized tankless can take so much electricity that it's hard to provision (at least, if other conservation measures aren't taken to reduce the needed GPM).
The hot water pipe between the water heater and the points-of-use will tend to go cold fairly quickly. As such, there's a "slug" of cold water that has to be pushed through the pipe before any hot water arrives. A centrally located tankless does nothing to improve this. However, with tankless heaters, it is possible to have multiple tankless heaters, each near a point-of-use (such as a bathroom). This eliminates the long pipe run and the waiting... and allows the local unit to be much smaller in capacity, since it only needs to support one "large" load of known size. For instance a bathroom's unit could be sized for its shower, and combined with a low-flow showerhead - making the unit size very practical. This would make the bathroom immune to temperature changes from use of hot water in other rooms (but not within the bathroom itself obviously).
An extreme of this is Great Britain's "electric shower" - an 8.5KW to 9.5KW (40A) unit that is directly attached to the shower.
The right combination of units could also reduce the overall ampacity needed, making the project more achievable.