Go ahead and put both loads on the same subpanel -- you don't need a relay for this, as the tankless heater isn't a continuous load (i.e. I can think of no situation where the domestic hot water should be running in a house for 3 hours straight!).
(The feeder calculations are as per 215.2(A)(1)(a) -- 125% of the 4500W storage heater is 5625W + the 5700W tankless heater yields 11325W, which is 47A@240VAC.)
Wow, very mysterious.
First of all, if the AC unit trips a breaker, it may be the compressor is intermittantly having a hard start. Mine was doing this in cooler weather, or at night. Adding a hard-start capacitor to the compressor fixed it.
But your event looks more troublesom than that, as you have unrelated circuits tripping together. What I would first guess is that you are having brown outs. The brown outs could be caused by the AC compressor having a hard-start. Or the brown-out could be caused by the grid voltage being low.
When a AC compressor starts, it acts like a dead-short for a brief moment, until the motor is spinning. The unit starts in the "locked rotor" condition, and the locked-rotor current is usually many times that of the breaker, so a unit that has a 40 amp breaker might have an locked rotor amp (LRA) rating of over 100amps. Most breakers trip on heat, so they can take a load 3 or 4 times higher than they are rated, for a few miliseconds.
So if your AC is slow to start, it might hit your house with a 100+ amp load for a second, causing your house voltage to drop. If the voltage drops, then many things you have plugged in with reactive loads will increase in current as the voltage drops out. The breakers run off current only, and don't care that the current is caused by a brown out, so if the brown-out causes current to exceed ratings, breakers trip.
So the first thing I would do is tell your AC repairman you are popping the AC breaker from time to time, and ask him to investigate putting a hard-start kit on your compressor. What the hard-start kit does it ads a little extra boost of start-up power to the compressor, so a tired old compressor can start easier, and avoid tripping the breaker.
Replacing the AC bereaker may be called for, but don't put one in that is larger than the name plate rating on the compressor. The compressor will say on the nameplate how large a breaker you should use. Don't put a larger one in, a larger breaker will not make a hard-start compressor better, it will make it burn-up quicker. The only fix is a new compressor or a hard-start kit, never a larger breaker. Of course that assumes it is your compressor. It could be something else entirely, like a power grid brownout.
In my case, the compressor hard-start was only when it was cool outside, but that is just one odd case, most of the time a hard start compressor will hard start all the time, or only when hot, or only when the unit is short cycles, or only when it's been off for a while. Your AC guy should know to look for hard-start when you mention the breaker tripping intermittently. You should hear him talk about trying a hard-start kit even if you don't mention it to him, but that doesn't mean you can't suggest it to him anyway... if he thinks you know what is going on, he is less inclined to give you B.S... If he is unscrupulous, he may suggest a compressor replacement first, but really a hard start kit is a better first move if a hard-start condition is suspected.
Best Answer
Normally you'd need the sub panel if for no other reason than to have a means to disconnect the sauna for maintenance. Since the sauna is in the same room you can go directly to it from the main panel as long as you have clear line of sight from the panel to the sauna. Your instructions show a sub panel and you shouldn't deviate from them but they also give wire sizes for going from main breaker to power unit (4#8,N & G) so I guess it's OK.
I think going to the sub panel would be cheaper.