Up until 2011 the National Electrical Code allowed a "3 wire" method of connecting a sub-panel. This means you have the two "hot" wires and a neutral running to the sub-panel as opposed to the "4 wire" method where you add a separate equipment ground wire.
So, when your house was built, it was perfectly legal to use the 3 wire method. There is no problem with this unless you connect a metal pathway, such as a water pipe, from one building to the other. This allows neutral current to flow on this parallel path. This could be potentially lethal to someone working on the pipe.
As to the main panel not having a neutral: it does. If you look again you will see the two hot wires and a bare conductor. There is no ground wire on the service drop. The bare conductor is the neutral. This is allowed on the service drop, but from that point on all neutral conductors must be insulated. Ground wires are always allowed to be bare.
So, in short, there is no problem with your system unless you run a metallic system to the sub-panel building. This would create a problem.
Big problem: no ground
NEC 2014 gives you broad latitude to share grounds among circuits. The one thing it does not allow you to do is share grounds among panels. Circuits served from the main house cannot use the grounding system from the subpanel, or vice versa.
Sharing neutral is out of the question
What has never been allowed is sharing a neutral among different circuits. This causes a lot of problems. It breaks GFCIs and AFCIs. It causes potential overloads, since neutrals do not have overcurrent protection (breakers) and depend on being mated with only one single hot which does have appropriate overcurrent protection.
What's more, neutrals have to be white. No tagging a colored wire. So if your three 12AWG wires do not have a white among them, you'd need to pull one - and might as well pull a ground too.
Of course... If the circuit was 240V, none of the wires would be neutral, would they?
Relays?
Given 3 colored wires, no neutral or ground, here's what makes sense to me. Power both circuits (the 3-way and the simple switched circuit) from the subpanel at the barn. Use the 3 wires to control the coils of two relays. I think if you put the relays immediately off the subpanel, e.g. in knockouts on the subpanel itself, you would effectively be inside the steel shielding of the subpanel and dodge the the grounding issue. It's a stretch, but it's all you got short of pulling more wires.
At the house, you bring 120 or 240V to your two switches. Plain 1-way switches. You send two switched wires and a common to the barn. Each relay coil takes 1 switched wire and shares the common.
Then you feed power off a breaker in the barn subpanel, which goes to the relay contact marked common on each relay. The relay to control the simple load is a plain SPST relay (any other kind would also do) and is wired like a plain switch. The common contact takes always-hot from the barn subpanel, the NO contact is switched-hot for that load.
The other relay is a SPDT type. Note that this is exactly the same layout as a 3-way switch, and you do the standard 3-way switch layout. You wire always-hot to the common relay contact. Each of the NO and NC contacts go to the two messengers, which go to the barn 3-way switch. The common of the barn 3-way switch goes to the light. Standard 3-way switch layout, except one switch is a relay.
That's how I would do it with the wires present.
Best Answer
I'm sorry, but we have to talk about legalities.
Especially given the high risk here.
You are not a licensed electrician and that places limits as to what you may do.
But people will assume the worst, especially after an accident.
The problem here is that between the 3-wire feed which may be 40 years old (but is that documented?), the known lost neutral (pretty much the worst-case scenario), and the pool for Pete's sake. Read electrical drowning reports, and they always open with scenes like this. You're already involved, so it's a liability albatross for you.
Job One must be to sever yourself from that liability, and the path there is to have a licensed electrician review the site, write an action plan, and serve it on the customer. That shifts liability to the customer and gets you off the hook. Preferably get the customer to pay for it, but even if you have to eat it, it's a "get out of jail free" card.
Don't take my word on it. Ask your lawyer.
Based on what you say, I'd confirm your analysis
Given that the voltage swings are extreme at the barn and pool, and not at the house, it seems like a lost neutral somewhere between house panel and barn panel. Given that several splices are involved in the transition from panels to pole line, it could be any of them.
You say that the installation was previously done (before your time) with 3-wire "triplex" overhead service drop cable, with a bare neutral. That was legal prior to (I want to say) 1999, but not in every case; where there were parallel metal things e.g. water pipe, the ground was sometimes required to keep fault current from corroding the pipes. Since it was installed prior to that, it should be grandfathered by the inspector. Not by the reaper, though: neutral problems like this one are exactly why it was outlawed.
So the short-term quick-fix is to find and fix that poor neutral connection. Note that NEC 2017 requires use of a torque wrench/screwdriver for most connections, but larger connections such as feeder required it prior to that.
Simply doing the quick fix and walking away will still leave you with big liability exposure.
The next things to look at
And again, merely looking at these things yourself will not reduce your liability.
But I would start with the Grounding Electrode System inclusive of Ufer grounds, ground rods, and all the special grounding and bonding that must happen around a pool. I'll bet you a lot of that is out-of-order even for the time of original installation. If work was illegal at the time it was done, it is not grandfathered and must be done to current Codes.
Then I'd look at GFCI requirements especially around the pool areas.
Next, I'd look at running that fourth ground wire. The problem with a 3-wire feeder is exactly this: nothing really controls a "lost neutral", and if the GES is in good order, it can actually float the earth around the buildings to dangerous levels. (which can combine rather badly with things like fence lines).
The electrician's report would cover all that.