[RPG] How to handle initiative when a new force joins a combat that’s already in progress

There are two main situations in which you could call for an initiative roll:

An enemy makes a hostile action

In the situation you described I'm not sure that drawing a sword would call for this. Instead, tell the players what he does and see how they react. If they say "I shoot him!", then you roll. If they say "I attempt diplomacy to calm the situation" then you don't.

• The enemy says something obviously threatening (e.g. "Prepare to die!" or "Guards, seize them!")
• The enemy goes to move (e.g. to flee, take cover, grab an item, charge, etc.)
• The enemy makes an attack (this may be a surprise round if you think it appropriate)

The PC makes a hostile action

Similar to above, this is anything for which the enemy would not just stand by and do nothing.

Here are some scenarios which might make this clear

Casandra is confronting a rival in a dark alley. It is not yet clear if there will be bloodshed or merely threats and a parting of ways.

1. Casandra runs her hand through her hair (no initiative roll).
2. Casandra runs her hand through her hair and flings a shuriken which was disguised as a hair clip (surprise round, followed by initiative roll).
3. Casandra says "If I see you in my territory again I'll cut your throat." (intimidation roll, but no initiative).
4. Casandra flees (initiative roll if her rival wishes to pursue).
5. The rival draws a dagger (initiative roll depending on how Casandra reacts, no surprise round).

You're right that the basic initiative rules just kinda assume everyone's in the fight at the beginning of the battle and don't say more than that, so the addition of late arrivals requires some interpretation.

In general, as new arrivals become aware of and desire to participate in the battle, they should just roll initiative for the first round they're aware and then be slotted into the existing order.

There are no "attacks of opportunity" specifically granted by initiative or surprise; I assume you are referring to the surprise round of free single actions that happens at the beginning of a combat when some combatants are unaware of others, and during which unaware PCs are flat-footed both for the surprise round and until they act on their first round. The rules are written to describe the surprise round as only being at the beginning of combat, but it's certainly realistic to say that new arrivals might be able to come upon current combatants more or less unawares, even with the "360 degree sight" Pathfinder affords everyone.

What I do is, if the new combatants just want to charge in and fight, I just slot them into the initiative. If, however, they do want to sneak up on the fight and take someone unawares, then I use the standard Stealth vs Perception check to determine awareness from the surprise rules. If someone is surprised, I don't insert a surprise round, but I do afford flat-footedness to their unaware opponents till they act.

Example

So for example, let's say Robilar (init 11) is fighting two orcs (inits 12 and 7). A pair of sleeping orcs, a shaman and a warrior, hear the fight in another room and go to join in. The warrior just runs into the fray, so he rolls initiative and gets a 13. At the top of the round, he runs in and chops at Robilar, with no special bonuses. The shaman decides to be sneaky and wants to peer through a door and snipe Robilar with a ray of exhaustion. He makes a Stealth check against Robilar's perception and wins, and rolls an init of 6. That's fine, Robilar goes on 11 but doesn't see him, and then the shaman zaps his flat-footed touch AC with a ray on 6, after which Robilar is quite aware of him (the orc doesn't have enough Stealth to try true sniper tricks with a hide at -20, so he just pushes the door the rest of the way open and fights normally).

In my opinion this merges rules compliance, speed of resolution, and realism of result.