Does the sub-panel seem over loaded? If so, I could keep the water-heater in the main panel and free up space in the panel another way.
Seems reasonable to me. Most of the equipment won't draw anywhere near the overcurrent rating, at least not during normal operation. Motor loads will draw a higher current on start, but you shouldn't have a problem.
I know I need four-strand wire to run to the sub-panel (2 hot, neutral, ground) but copper or aluminum and what gauge?
You can use either copper or aluminum, though I recommend copper for DIYers. Copper is quite a bit more expensive, but it's easier to work with (in my opinion). If you feel confident working with aluminum conductors, you can save some money using it.
I've covered the topic of feeder sizing here, so I won't go into detail. If you're using copper, you'll want to use 3 AWG conductors. If you choose to use aluminum, you'll need 1 AWG conductors.
If you want to run a single cable, instead of individual conductors in conduit. You can buy what's called 3-3-3-5 SER cable (1-1-1-3 for aluminum), which will contain three 3 AWG conductors (hot,hot,neutral) and a 5 AWG grounding conductor.
When I run the wire along the floor joist, does it need to be secured to the joist or can it just hang there and rest on the drop ceiling? Seems like it should be secured to the joist with wire hanger or something.
You'll have to attach the cable to the joists, using 1 - 1 1/4" staples or other approved means. Check the packaging, to make sure they are rated for the size cable you're using.
What are the things about this project that I don't know that I don't know. :) These are the scary things IMO...i.e. the questions I don't know enough to ask.
The cable you'll be working with is thick and heavy, and it's not going to be fun pulling it. You'll probably want a couple helpers, to help you wrangle it.
Make sure all your connections are tightened to the manufacturer's specified torque.
If you choose aluminum conductors, make doubly sure you tighten the connections. And don't forget the anti-oxidant.
Come back a day or two after the panel has been put into service, and tighten any connections that need it.
Don't forget to remove the bonding jumper between the grounded and grounding bus bars.
You'll need clamps big enough for the cable, to secure it to the panels.
should I put a 100 amp breaker in the sub-panel to act as the "main" for the sub-panel? Or is the 100 amp breaker in the main panel sufficient?
You can usually pick up a main breaker panel, for about the same price as a main lug only (MLO) panel. In my opinion, unless the secondary panel is next to; or within sight of, the main panel. You're better served to install a main breaker panel. It simply offers better protection during maintenance, or other work within the panel.
For example. If you turn off the feeder breaker in the main panel, and start working in the secondary panel. Somebody could easily come along, and flip on the feeder breaker. Since you can't keep an eye on the breaker, you can never be sure the panel will be dead. (unless of course you're using a lockout like you should).
If the secondary panel is in a separate building or structure, then you either need a main breaker, a main disconnect, or the ability to disconnect all ungrounded conductors within 6 or less hand moves.
Lucky you
Your panels are set up very advantageously to put in a generator transfer switch. All the loads you want to switch (but one) are already in a subpanel. This makes this super easy. It's almost like somebody planned it... Except the guy who put the pump circuit in the main panel did not get the memo.
You need to put a different transfer switch in between the two boxes. There are a couple ways to do this.
One is fit the subpanel with a manual interlock switch. This is a listed (offically tested) modification to the subpanel where it puts two breakers opposite from each other with a sliding plate so they can't both be on. Simple and cheap, and you may even be able to retrofit your existing panel. Must be thrown manually.
The other way is to pick a spot along the cable between the panels (or elsewhere if you don't mind rerouting cable), cut the cable, and insert the transfer switch inline. This will work with any kind of transfer switch, including automatic. You'll need to cut the cable on one side with enough slack to work, and the other side will be too short so you will have to replace that "half" of the run. So choose location and cut point very carefully to your advantage.
That one breaker in the wrong panel
The stuff you are imagining, you cannot do anything like that. In that approach, the options are break the law, backfeed the grid and kill linemen; or spend a king's ransom on more heavy cables than the underside of an NYC subway car, in a veritable Gordian knot of transfer switch wiring that nobody will be able to figure out after the fact. And the power company and inspector will absolutely hate it.
The right way is easy, if annoying: move the circuit to the subpanel. Extend the pump circuit, all wires, to the subpanel and land it on a double breaker, ground bar, and neutral if used. Do not continue to use the ground in the main panel, in fact if you make this splice inside the main panel, tape the ground wire with green tape to insulate it from the main panel. It's not the end of the world if it grounds accidentally, but grounds must go to the same panel the hots do.
I know that's a pain, but it's way less of a pain than anything else you could do legally.
Not room in the subpanel?
This happens a lot, some guy drove back from the subpanel store slapping himself on the back for saving $30... by buying barely enough spaces. Don't be that guy. You're going to need 6 right off the bat - 4 for the transfer switch and 2 for the well pump.
Worst case you may need to replace the subpanel, and fortujately that is DIY-possible because you can entirely shut off the panel at the main breaker. Also a great time to be looking for transfer switch friendly panels if you want to go that way. Don't scrimp - slap yourself on the back for buying twice as many spaces as you need today. The bigger panels often come with "bonus breakers" which save you some money too.
Best Answer
There is no rule against having multiple breakers in the same circuit. The drawbacks are only the wasted money and more things to troubleshoot when something goes awry, i.e. the well pump stops working and now you have 3 breakers to check instead of one. But you have a slightly different situation after you posted this:
What that did was change up the definition of your circuit from being a "branch" to being a "feeder". A branch circuit is one that is the LAST circuit before the load. If you had ONLY the well pump at the very end, then the one breaker in the panel at the house was the only one necessary. The one out near the pump could have been a non-fused disconnect switch within sight of the well. The same could have been true of the "Reliance" box circuit as well (although most likely they only come with breakers).
But by having the electrician add the 110V circuit at the well head, you now have TWO load circuits, so you need TWO branch protective devices there, and the breaker in your house panel is now the feeder for those branches. It's not a problem regarding the size, either way they are just protecting the conductors so as long as your feeder conductors are at least #10 you are fine, as well as your branch conductors for the pump itself. But now your 110V circuit must have another branch protective device. Assuming there was a good reason for the transformer (i.e. no neutral conductor run out to the well), you must have some sort of branch protective device for the primary of that transformer at the very least. Hopefully there is. Could be fuses, could be another smaller 2 pole breaker, but there must be something there protecting that transformer.