The answer depends on the details of what loads are already connected to your 150A panel. You didn't say if this was a residence or other. Certainly a second 3-phase service would be highly unusual in a residence.
However, assuming that the sum of the existing loads on the 150A panel don't preclude adding the additional load of the guest house, then the 4AWG aluminum conductors would admit a maximum 60A feed (Unless you can find a 70A breaker… I've never seen one). Not 80A.
As to which panel to connect… if there are existing circuits in the guest house, I would be leery of connecting the SER to a different panel than these circuits… mixing circuits from two different services in the same dwelling is increasing the chance for disastrous errors in the future.
And unless you're really experienced, knowledgable, and qualified about electric power distribution, I would advise that you not touch the 3-phase stuff. There are numerous non-obvious pitfalls in that for the untrained person.
Spaces vs Circuits
First let's explain the 8/16 thing, or "8 spaces but 16 circuits". At some point, panel manufacturers developed what I (alone) call a "double-stuff" breaker, which crams 2 breakers into a single space. This is intended to solve crowding problems in too-small panels.
photo source: Home Depot
Relying on these, marketing called their boxes 2X-circuit (X-space). Around 2014 they mysteriously stopped doing that. Recent NEC changes required AFCI or GFCI breakers on damn near every circuit... AFCI and GFCI are not made in double-stuff breakers.
Your 16/8 service panel is a legacy of that bad labeling. You have an 8-space panel, intended to replace an 8-space fuse box.
Panel space is dirt cheap. Go nuts.
You are dreaming of a 16-space panel. And even that is dreadfully small and you will find it rather limiting even with 100A service.
The practice of many electricians is to give you the smallest panel possible. That's first, to force you into more subpanel work if you expand further (more money for them) and second to save themselves $20 in part cost.
The cost of a larger panel is trivial tiny compared to total project cost. It is very correct to be extreme. Please spend an extra $100 for a 42-space panel, unless you just can't make it fit. Honestly I'd go 60-space. Not least, those larger panels are 225A-ready.
You never know. You might do a kitchen remodel and go hey, I'd like a separate circuit for refrigerator done microwave done dishwasher done disposal done 3 outlet circuits done. You have the space.
When you buy a PC, do you really calculate your needed hard drive space and buy just what you need... Or do you just buy the big 3TB drive? Of course you do. Same thing here.
More work to do at the meter
OK so the power company says their side of the cable is 320A. If they say so.
They say their meter is 320A. If they say so.
But your meter housing is not 320A.
To support more than the listed 125A, you will need to upgrade your meter housing. There's no question of that. That's your equipment and you have to pay for it, probably, unless your power company does something different with cost sharing.
You won't be able to replace parts of it, you will probably even have to replace both sides of it as a single unit. However it is very nice to have your main breaker in the meter. It means you can fully de-energize your main panel, which makes it safer for you to do yourself.
The 320A may be shared
Since it's multi-unit, the power company provisioning 320A makes more sense. Trouble is, this 320A service is probably shared between both units. So it's not as much as you think.
You really need to talk to your power company about what they consider this "320A" to be. It may be a relabeling of what other power companies call 400A service. So they may be willing to feed two 200A panels, or 100/100/200. This is a conversation you can have only with them.
More food for thought is sub-metering units, and separately metering common spaces. The new thing in rental properties is to have one main meter and the landlord has sub-meters per tenant. Many landlords prefer (and some law requires) commons space usage to be on a separate (sub) meter, i.e. heating, yard and commons lighting, anti-freeze pipe wrap or roof/gutter heat, coin laundry, Christmas decorations etc.
Best Answer
Those two 50A single breakers need to be handle-tied, or replaced with a 50A 2-pole breaker. You can't have two singles on a 240V load like that.
The empty breaker space at bottom left should be filled with a proper UL-listed thing. They make blank filler plates, but I find them flimsy and expensive. I just use actual breakers, a CH120 is around $5.
Yeah, that plan is fine. Use "MAC Block connectors" to splice from the #6 aluminum to whatever other wires (Al or Cu) you'd continue with.
Note that since the #6 aluminum run does not have a neutral, they must be 240V-only heat pumps. Cannot use bare as a neutral.
You may find it challenging to land #6 wire on a 25A breaker.
Waii--what???
OK, you see that dual 30A and you immediately get "This is a 30A 240V circuit".
But then you see those two 50A singles and for some reason you think that isn't the same exact thing. It is the same exact thing, but somebody left off the mandatory handle tie so it looks weird.
So think of it as a 50A 2-pole breaker.
The problem is, the existing #4Al cable has only 3 wires and cannot supply a 120/240V subpanel.
The good news is, the SE/XHHW cable is allowed 75C temperature, so it can be provisioned to 65A and use a 70A feed breaker. There is no such thing as a 70A/1-pole breaker, so you will need to either re-use a 50A, or obtain a 70A/2-pole and use only one pole of it.