[RPG] How to stat up and judge the challenge of supernatural NPCs

You're right that the basic initiative rules just kinda assume everyone's in the fight at the beginning of the battle and don't say more than that, so the addition of late arrivals requires some interpretation.

In general, as new arrivals become aware of and desire to participate in the battle, they should just roll initiative for the first round they're aware and then be slotted into the existing order.

There are no "attacks of opportunity" specifically granted by initiative or surprise; I assume you are referring to the surprise round of free single actions that happens at the beginning of a combat when some combatants are unaware of others, and during which unaware PCs are flat-footed both for the surprise round and until they act on their first round. The rules are written to describe the surprise round as only being at the beginning of combat, but it's certainly realistic to say that new arrivals might be able to come upon current combatants more or less unawares, even with the "360 degree sight" Pathfinder affords everyone.

What I do is, if the new combatants just want to charge in and fight, I just slot them into the initiative. If, however, they do want to sneak up on the fight and take someone unawares, then I use the standard Stealth vs Perception check to determine awareness from the surprise rules. If someone is surprised, I don't insert a surprise round, but I do afford flat-footedness to their unaware opponents till they act.

Example

So for example, let's say Robilar (init 11) is fighting two orcs (inits 12 and 7). A pair of sleeping orcs, a shaman and a warrior, hear the fight in another room and go to join in. The warrior just runs into the fray, so he rolls initiative and gets a 13. At the top of the round, he runs in and chops at Robilar, with no special bonuses. The shaman decides to be sneaky and wants to peer through a door and snipe Robilar with a ray of exhaustion. He makes a Stealth check against Robilar's perception and wins, and rolls an init of 6. That's fine, Robilar goes on 11 but doesn't see him, and then the shaman zaps his flat-footed touch AC with a ray on 6, after which Robilar is quite aware of him (the orc doesn't have enough Stealth to try true sniper tricks with a hide at -20, so he just pushes the door the rest of the way open and fights normally).

In my opinion this merges rules compliance, speed of resolution, and realism of result.

A group 3rd- to 5th-level PCs in B/X and compatible games can defeat NPCs 2 to 3 levels higher than the PCs, and maybe even NPCs of higher levels than that if the PCs scout ahead, plan the engagement, possess the necessary equipment, and have the element of surprise. Remember that one-shot magic items such as scrolls and potions can vastly amplify the party's capabilities!

Obviously, in a straight-up fight, the PCs would surely lose (except at my table where I always roll poorly for bosses yet exceptionally well for their weak underlings). PCs should be given fair warning about the possibility of their own defeat. If the PCs can't knock down the NPC quickly, the NPCs will retaliate, and that could result in the death of a PC or two, but that's possible against evenly matched foes, so it's not really that big of a deal.