Does the sub-panel seem over loaded? If so, I could keep the water-heater in the main panel and free up space in the panel another way.
Seems reasonable to me. Most of the equipment won't draw anywhere near the overcurrent rating, at least not during normal operation. Motor loads will draw a higher current on start, but you shouldn't have a problem.
I know I need four-strand wire to run to the sub-panel (2 hot, neutral, ground) but copper or aluminum and what gauge?
You can use either copper or aluminum, though I recommend copper for DIYers. Copper is quite a bit more expensive, but it's easier to work with (in my opinion). If you feel confident working with aluminum conductors, you can save some money using it.
I've covered the topic of feeder sizing here, so I won't go into detail. If you're using copper, you'll want to use 3 AWG conductors. If you choose to use aluminum, you'll need 1 AWG conductors.
If you want to run a single cable, instead of individual conductors in conduit. You can buy what's called 3-3-3-5 SER cable (1-1-1-3 for aluminum), which will contain three 3 AWG conductors (hot,hot,neutral) and a 5 AWG grounding conductor.
When I run the wire along the floor joist, does it need to be secured to the joist or can it just hang there and rest on the drop ceiling? Seems like it should be secured to the joist with wire hanger or something.
You'll have to attach the cable to the joists, using 1 - 1 1/4" staples or other approved means. Check the packaging, to make sure they are rated for the size cable you're using.
What are the things about this project that I don't know that I don't know. :) These are the scary things IMO...i.e. the questions I don't know enough to ask.
The cable you'll be working with is thick and heavy, and it's not going to be fun pulling it. You'll probably want a couple helpers, to help you wrangle it.
Make sure all your connections are tightened to the manufacturer's specified torque.
If you choose aluminum conductors, make doubly sure you tighten the connections. And don't forget the anti-oxidant.
Come back a day or two after the panel has been put into service, and tighten any connections that need it.
Don't forget to remove the bonding jumper between the grounded and grounding bus bars.
You'll need clamps big enough for the cable, to secure it to the panels.
should I put a 100 amp breaker in the sub-panel to act as the "main" for the sub-panel? Or is the 100 amp breaker in the main panel sufficient?
You can usually pick up a main breaker panel, for about the same price as a main lug only (MLO) panel. In my opinion, unless the secondary panel is next to; or within sight of, the main panel. You're better served to install a main breaker panel. It simply offers better protection during maintenance, or other work within the panel.
For example. If you turn off the feeder breaker in the main panel, and start working in the secondary panel. Somebody could easily come along, and flip on the feeder breaker. Since you can't keep an eye on the breaker, you can never be sure the panel will be dead. (unless of course you're using a lockout like you should).
If the secondary panel is in a separate building or structure, then you either need a main breaker, a main disconnect, or the ability to disconnect all ungrounded conductors within 6 or less hand moves.
Best Answer
225A just isn't quite enough for everything you want to do here
Normally, I do not advocate for 400A services to single dwelling units; most of the drivers for such are large single loads such as whole-house instantaneous water heaters and oversized backup strip heating, and these tend to be harsh on the power grid. The next smaller service size SCE provides, though, is 225A, and it just isn't quite enough for all you wish to do here, between the two hot water heaters, the heatpump, the EV charger, and the shop. Even without the shop taken into account, I get a service requirement of 212A (assuming a 23A@240V heat pump condenser, 9.6kVA for the EVSE, and 5 kitchen small appliance branch circuits as well as an 8kVA range allowance). Given that a woodshop would have a multi-horsepower dust collector in it atop other motor loads, and other sorts of shop space are equally power hungry, this points towards 400A service, or making other sacrifices (such as ditching the second hot water heater).
Fitting it all in
One way to make room for the shop in your setup would be to sacrifice the second hot water heater. While it may sound like a terrible problem to provision hot water throughout a 3800 ft2 house from a single point, there are tricks that can be used to ease this, and one of the most significant is to insulate, or lag, the hot water piping throughout the house, to at least R-6, using standard foam pipe insulation parts.
This keeps the water in the pipes from cooling off nearly as quickly, allowing near-instant warm-to-hot water without the need for an energy-hogging recirculation setup or multiple hot water heaters scattered across the house. It also reduces standby losses from the heater itself, which means that lagging the hot water plumbing is a good idea in any case. However, if the demand is too high for a single hot water tank, that is also an understandable reason to have multiple hot water heaters, so I can understand if this is not an option for your situation.
If one is able to do that, then one can use a 225A, EUSERC compliant, over-under meter-loadcenter as a distribution panel to feed the subpanels at each end of the house, as well as the battery/solar system and its associated panel. While this hardware is uncommon, it does exist; the Siemens MC3042B1225SED is probably the easiest-to-find example. The other downside of this cram job is that it would limit you to a dust collector (of about 2-3 HP) + one power tool (1500VA) running for a woodshop.
Going for the big amps
If you do decide to go with the dual hot water heaters, though, then you likely will need a 400A service and matching equipment in order to provide sufficient headroom for shop tools and future expansion. For the meter main, you can take one of two routes; either you can use a 400A meter-loadcenter, such as a Siemens MC2442B1400SD(S), as a distribution panel with a single main disconnect and feeder breakers to the aforementioned subpanels, or you can use a 400A multiple-disconnect meter-main, namely the Square-D CU12L400CB/CN, with the two 200A mains feeding the subpanels at each end of the house and a breaker in the loadcenter section feeding the solar/battery system.
As to those subpanels
Either way, you'll want to go with a pair of 42-space, 200A or 225A, main lug panels for the main subpanels in your setup. This will provide ample expansion space for the future, compared to the pair of 30-space panels you proposed, without costing much more at all in the grand scheme of things.
You will also want a 24-space or 30-space, 125A, main lug panel as a dedicated subpanel for standby loads, given that you are taking about a multimode (battery backed) solar system vs. a simple grid-tied setup. While it generally does not make much sense to have batteries without the ability to disconnect and run at least some loads off-grid, especially with the volatility of the modern grid, it also does not make sense to have a standby system, whether it be a battery system or a generator, try to run the whole house, either.
What about that solar and battery stuff?
While the combination of solar and storage is certainly not a bad proposition, it does raise some questions about choices in storage technology and system architecture, as getting the maximum utility out of battery storage is not merely a matter of hanging an "AC battery" off a branch circuit and calling it a day.
The first key decision you have to make with solar+storage is how you want the two to integrate, and there are three ways to do that: low-voltage DC coupling, high-voltage DC-coupling, and AC-coupling. Each of these has pros and cons, which I will cover here in turn.
Going old school
The classical solution to the problem of integrating solar and storage together is to follow the lead of the off-grid folks, using a DC-coupled system running at low voltage, typically 48VDC nominal. These systems are designed around an open architecture, with a variety of manufacturers making equipment suited for these applications alongside a free choice of battery chemistry, with classical valve-regulated lead-acid technology providing a low first-cost option, large-capacity lithium-ion storage solutions offering a balanced package, and modular lithium-iron-phosphate battery systems providing a high-performance solution at the cost of a high upfront pricetag. They also provide good battery-charging control and low conversion losses, which is useful in places where the grid is relatively intermittent as it maximizes the practical utility of the solar harvest available.
This approach is not without its downsides, though. First off, these systems require quite a bit of balance of system equipment, with separate charge controllers, battery/multimode inverters, and relatively costly heavy-duty DC switchgear, as well as rapid shutdown and arc fault protection devices. Atop that cost, this approach requires fat copper wiring for the low-voltage DC bus to keep voltage drops acceptable, and fat copper is not cheap, especially in the fine-stranded type required sometimes for battery and inverter hookups. The final downside is that these systems require an installer well-versed in the off-grid/multimode world; "box-pusher" contractors that are only familiar with a single manufacturer's hardware will struggle with the system integration required in such a setup.
Cranking up the juice (but it's not that simple)
The second approach is the one taken by SolarEdge, Pika Energy, and a couple other vendors, and that involves using a vendor-specific high-voltage DC battery pack alongside a specially equipped high-voltage string inverter. These systems provide tight integration due to their single-vendor sourcing, while providing low standby losses and good charge control just like their low-voltage counterparts, and have the advantage that all the troublesome high-voltage DC switchgear is integrated into factory-built equipment. However, getting replacement parts and upgrades are more difficult due to these systems having a proprietary architecture, and you are also locked into lithium-ion chemistries with this approach.
For a completely different solution...
The other alternative would be to use an AC coupled multimode solar system. This uses either standard string inverters or microinverters to convert the solar harvest to AC, then feeds that AC to a dedicated solar or standby bus. On this bus either sits a multimode/battery inverter's standby-side output, or an AC battery storage system such as the current-generation Tesla Powerwalls, that takes the AC from the solar inverters and converts it back to DC for battery storage. There will also be a transfer device (either a dedicated unit if an integrated AC ESS is used, or built into the aforementioned multimode inverter) that isolates your standby system from the rest of the AC grid during a power outage.
The primary advantage this has is that it is easy to install, using relatively thin wires and standard AC distribution equipment for most of the system, while providing the opportunity for an open-architecture system to be deployed, easing hardware upgrades and replacements down the line. However, battery charge control is much more difficult on these systems, requiring a certain degree of configuration intelligence from both the battery and solar inverters, and there are extra conversion losses in these systems compared to their DC-coupled counterparts. Furthermore, taking full advantage of the open-architecture possibilities that using a multimode inverter in an AC-coupled system gives requires some of that aforementioned heavy-duty DC switchgear and thick cabling, and it also limits the amount of solar one can install so that the battery inverter can regulate the standby power system during outages.
What should go on the standby system?
The other hard decision one has to make in this case is the matter of what should be powered by your standby system vs. what you can afford to lose when the grid goes dark. In your case, you have one piece of good news and one piece of bad news. The good news is that since you are in a warm enough climate that pipe freezups are not a serious concern, losing your heat to a power outage is an acceptable state of affairs. The bad news is that unitary/hybrid heat pump water heaters, especially when your house has two of them, draw too much power for effective standby operation, which means that a power outage makes for some rather chilly showers.
That being said, if you are not dealing with extended outages, we can rule several other things out as standby loads. The EVSE, being both a high-power load and something that can be deferred for the span of a shorter outage, has no need to be on the standby bus, for instance; nor does the range (far too large of a load), or any laundry equipment (since it's not running often enough to matter during all but the longest of outages). Furthermore, the range is simply too large to run on any reasonably sized standby system, and the heat pump itself is a heavy starting-current load, atop being non-essential in a relatively mild climate.
This leaves us with basic lighting, refrigeration, and a small selection of general receptacles (for essentials such as charging a cell phone, or keeping clocks running), as well as the nicety of a small-appliance branch circuit that can be used to run a microwave, toaster oven, or electric griddle, permitting a modicum of cooking capability during a power outage. Fortunately, with the advent of LED lighting, one probably only needs to provide a few kW of standby power to run all this, and that eases things considerably, allowing a not-totally-unreasonable battery to provide a day or two of autonomy, even if the grid is down and the solar harvest empty.
Other notes
If you use a multimode inverter, whether DC or AC coupled, you'll want to provide a bypass facility so that servicing the inverter can be done safely without killing power to the standby loads; this can be done using a bog-standard safety switch as a dedicated inverter disconnect (something that your utility may already require), with power tapped off the line-side of the inverter disconnect to feed a breaker in the standby panel that is interlocked with the standby panel's inverter breaker using an ordinary generator interlock kit.
Last, but not least, you or your installer will need to torque all set-screw type connections to manufacturer's specifications using an inch-pound torque wrench or torque screwdriver. This is required by 2017 NEC 110.14(D), and is good practice in any case, lest your electrical system lose you the race!