Wire size for a 150 amp sub panel from a 150 amp breaker added to a 400 amp main panel

The answer depends on the details of what loads are already connected to your 150A panel. You didn't say if this was a residence or other. Certainly a second 3-phase service would be highly unusual in a residence.

However, assuming that the sum of the existing loads on the 150A panel don't preclude adding the additional load of the guest house, then the 4AWG aluminum conductors would admit a maximum 60A feed (Unless you can find a 70A breaker… I've never seen one). Not 80A.

As to which panel to connect… if there are existing circuits in the guest house, I would be leery of connecting the SER to a different panel than these circuits… mixing circuits from two different services in the same dwelling is increasing the chance for disastrous errors in the future.

And unless you're really experienced, knowledgable, and qualified about electric power distribution, I would advise that you not touch the 3-phase stuff. There are numerous non-obvious pitfalls in that for the untrained person.

Neutral and ground being separated means you must remove the neutral-ground bonding device (commonly, a screw) from the neutral bar. You must also have a separate ground bar, which the cheapie big-box panels don't always come with.

You absolutely need both local grounding rods and a ground wire from the main panel. They do different jobs. Remember, electricity travels in loops and wants to return to source. The local grounding rod returns ESD and lightning, which is sourced from earth and should be returned right at the shed, not carried 150' on #6 wire to your house. Human electricity is sourced from the neutral bus on your transformer; the ground wire returns fault current to the neutral in your supply at the main panel. Dirt doesn't conduct well enough to do this job, all you will do is electrify all the grounds in your building.

Since you are doing a pole line, you really ought to use a "messenger wire" optimized for carrying the physical weight of the wire. This messenger can double as the ground wire, in fact you will want to earth it!

You are correct that a subpanel main breaker is simply a shutoff switch, which is mandatory in outbuildings. Its ampacity does not matter at all. Playing games with breaker sizes to try to get the nearer breaker to trip first, doesn't really work.

If you use aluminum wire use #4Al instead of #6Cu. They have the exact same current capacities and you will not need to change your calculations at all.

Also stop buying at big-box stores when you get as sophisticated as a pole line. Go to a real electrical supply, they have what you need in stock and they know what they're talking about.