The biggest key to creating interesting boss fights (in my experience) is to introduce an element of surprise or guess work. Fights are boring if they're just constant dice rolls back and forth where everything goes as expected. But you can make mechanics which keep the players guessing and on their toes, which force them to constantly be thinking about what's going on or what to do next.
For example, in the culmination to one of my campaigns, I had the final boss have a "shifting immunity". Each round, I would roll on a table to see what type of energy or effect he was immune to that round, and each immunity had an associated color. So each round, the boss would change colors and have different things it was vulnerable to.
This forced the players to be paying attention and didn't allow them to just sit back and spam the same moves over and over. ("What? That fireball worked last time...") It also made it interesting for them, because, in addition to being a shifting immunity, it would grant a different power to the boss.
That's the basic premise. If you want a fight to be long, keep it interesting with different mechanics. This can be something like I've described above, or a "phased" fight (i.e., the boss has several different stages or modes), or something like a very specific effect he/she/it is vulnerable to for a killing blow. It forces you to get creative, but there's no way around that; if you just rely on what's in the source books, well... the players have read those just like you have.
You're right that the basic initiative rules just kinda assume everyone's in the fight at the beginning of the battle and don't say more than that, so the addition of late arrivals requires some interpretation.
In general, as new arrivals become aware of and desire to participate in the battle, they should just roll initiative for the first round they're aware and then be slotted into the existing order.
There are no "attacks of opportunity" specifically granted by initiative or surprise; I assume you are referring to the surprise round of free single actions that happens at the beginning of a combat when some combatants are unaware of others, and during which unaware PCs are flat-footed both for the surprise round and until they act on their first round. The rules are written to describe the surprise round as only being at the beginning of combat, but it's certainly realistic to say that new arrivals might be able to come upon current combatants more or less unawares, even with the "360 degree sight" Pathfinder affords everyone.
What I do is, if the new combatants just want to charge in and fight, I just slot them into the initiative. If, however, they do want to sneak up on the fight and take someone unawares, then I use the standard Stealth vs Perception check to determine awareness from the surprise rules. If someone is surprised, I don't insert a surprise round, but I do afford flat-footedness to their unaware opponents till they act.
Example
So for example, let's say Robilar (init 11) is fighting two orcs (inits 12 and 7). A pair of sleeping orcs, a shaman and a warrior, hear the fight in another room and go to join in. The warrior just runs into the fray, so he rolls initiative and gets a 13. At the top of the round, he runs in and chops at Robilar, with no special bonuses. The shaman decides to be sneaky and wants to peer through a door and snipe Robilar with a ray of exhaustion. He makes a Stealth check against Robilar's perception and wins, and rolls an init of 6. That's fine, Robilar goes on 11 but doesn't see him, and then the shaman zaps his flat-footed touch AC with a ray on 6, after which Robilar is quite aware of him (the orc doesn't have enough Stealth to try true sniper tricks with a hide at -20, so he just pushes the door the rest of the way open and fights normally).
In my opinion this merges rules compliance, speed of resolution, and realism of result.
Best Answer
Demonstrate it to the PCs
The key problem with these complicated mechanics is that it's very difficult to telegraph them to your players. Video games can get away with complicated mechanics because they can highlight weak points, and the full range of actions a player can accomplish are limited by the game. In an RPG, where players can attempt anything they can think of, the range of possibilities is enormous and insurmountable, especially when there's a lich trying to kill them.
If you don't want to tell the PCs outright, with an NPC or some other clue, you could demonstrate the weakness. Have the Lich be noticeably afraid of bones, and emphasize his caution around his minion's bones. In another case, you could show a minor accident that causes an outsize amount of pain to the boss, like dripping water on a fire-based creature or something.
It shouldn't be hard.
When you're building a puzzle like this, it's easy to be too subtle about the clues. Remember that the PCs are receiving a constant stream of information from you, some of which might not be relevant. Unless you want them to waste time obsessively scrutinizing everything you say, you should leave multiple hints that seem obvious to you. After all, the negative implications of it being too easy are pretty small compared to the TPK that happens when it's too hard. For example, bosses in the Legend of Zelda series almost always have a big, colorful weak point that's obvious in their introduction cutscene.
The stakes and the difficulty are inversely correlated
In the Lich scenario, the PCs are actively being killed, and don't have much time to spare. If they waste a few rounds doing something useless, the consequences are dire for them. Therefore, the puzzle should be something they can figure out in one or two tries, because they might not have more than that.
On the other hand, if the Lich is already dead and the PCs have his phylactery, the stakes are much lower. Nobody's trying to kill them, and the PCs have a few days to figure out how to destroy it. In this case, you can make the puzzle a lot more complex and subtle, since your players can afford to spend time trying different things.
In the middle lies a weak boss, who can't deal too much damage. If you tone down the power of the boss significantly, so that neither side can easily defeat the other, you open more space for the characters to work out a puzzle boss while still keeping some pressure on the PCs. Of course, you'd have to make the puzzle interesting, or else it just becomes a big slog.