The combat begins when the Darkmantle decides to attack. If the players don't notice the Darkmantle at the beginning of combat, they are surprised:
Any character or monster that doesn’t notice a threat is surprised.
So, thanks to False Appearance, the Darkmantle has a Surprise attack. Also, when the combat starts, the adventurers don't know that they're in a combat situation and so are not scanning the entire space all the time, unless previously stated (they are not combat-aware), so the Darkmantle remains hidden until its turn. However, all characters that are higher in the initiative order than the Darkmantle are no longer surprised at the end of their turn (or, better say, the "Surprise" property is no longer applied to them)! They are thus able to take reactions. For reference, this answer to another question (also linked in the comment of this question) breaks it down in full detail.
The Unseen Attacker advantage, however, depends on the DM's ruling, as the RAW are not specific about this.
As far as Darkmantles go, they could approach their target unnoticed. They don't have to fly, they can just drop down. One could reasonably fall unnoticed as easily as dropping a curtain on top of someone. Even if someone would notice the creature as it was falling, they'd only get enough time to look up and see the horrid array of teeth approaching swiftly.
If you still believe your Darkmantles are unable to automatically Batman your PC's, then a Stealth check is appropriate, and should be made during the Darkmantle's Move action. In either case, the Surprise Attack persists, it is only the Unseen Attacker advantage that is at stake.
Keep in mind that the target of the attack must be taken into account. If the Darkmantle attacks a character that was higher in the initiative order, the character can take reactions (like Tempest Cleric's Wrath of the Storm), regardless of whether the Darkmantle remained hidden during its movement or not. The very act of attacking reveals it:
If you are hidden—both unseen and unheard—when you make an attack, you give away your location when the attack hits or misses.
However, if the trigger for their reaction is before the attack, the Darkmantle's visibility has to be taken into account.
If the Darkmantle attacks a character that's lower in the initiative order, the character won't be able to react until the end of its own turn. So, even if his reactions trigger "on hit", he won't be able to use them. However, if the characters that went before the Darkmantle have a reaction that triggers "when an ally is attacked", they can take that reaction (again, because the attack reveals it).
There is only the final, special case to consider, where the Stealth check is only partially successful and one or more characters notice the Darkmantle as it moves in to strike. It is fully within their ability to warn the others of the threat, as they can talk out of turn (within reason). However, as it's the Darkmantle's movement that triggers the Stealth check, the players that were surprised at the moment of the attack still remain surprised. Whether the darkmantle retains its Unseen Attacker advantage or not is up to the DM and his opinion on how fast a character can react to a warning.
From a role-playing perspective, if the Darkmantle remains unseen the adventurers either didn't notice the attack until the moment it hit, or they noticed it too late to do anything about it (including removing the advantage).
Here's an example of how that could be acted out:
(A PC walks under the Darkmantle and it decides to attack.)
DM: As you explore the dark cave, you feel a slight chill up your spine... Roll Initiative!
Player: Why? Do we see anyone here?
DM: Not yet.
(Initiative order is set. )
DM (to the players at the top of the initiative order, before the Darkmantle): You are not aware of any threats and proceed as you were.
(The Darkmantle's turn comes. It makes its optional Stealth check... and remains hidden. It swoops down on its unsuspecting target and rolls its attack with advantage... success!)
DM: Alright... Aldore, your head is suddenly wrapped in a curtain of flesh and you feel piercing teeth gnawing at your cranium. You take XX damage!
(DM now describes the attacker, and the battle continues. The characters that haven't had their turns yet are still surprised. When all turns resolve, the combat continues as usual from the top of the initiative order.)
Now, if the creature missed the attack (either by lacking or despite the advantage) you could have it be because the character noticed it and dodged in the last second, or you could make the Darkmantle comically flop on the ground in the middle of the party. Your choice :)
Basic
I'm going to address the general case rather than the highly specific example in the question. General guidance will be more useful to other readers with a similar question.
That said, let's start with the rules text...
If neither side tries to be stealthy, they automatically notice each other. Otherwise, the DM compares the Dexterity (Stealth) checks of anyone hiding with the passive Wisdom (Perception) score of each creature on the opposing side. Any character or monster that doesn’t notice a threat is surprised at the start of the counter.
By definition, the whole group of player characters are trying to surprise the monsters, or none of them are. The less-capable ones can remain stationary and further away to facilitate that surprise, but they're still involved and still need to roll. The surprised/not surprised status is determined before initiative is rolled.
The reality is that the entire party is trying to be stealthy - some are just trying to do it while moving. The thunky rear guard group still needs to make Dexterity (Stealth) checks, but there are some modifiers that should be applied - advantage and disadvantage.
- The complete lack of motion could grant the folks hanging behind advantage on their checks - in all likelihood simply washing with the disadvantage from the armor they're probably wearing.
- The distance and obstructions could impose disadvantage on the passive Perception of the enemies, but only against the rear guard and only if they're far enough back that the distance is a hindrance to the targets. If they're just outside the room, no disadvantage should apply. If they're down the hall or around a corner, then it just might.
In effect, the advance team rolls Stealth with a DC of the targets' Passive Perception. The rear guard rolls Stealth with advantage against a DC equal to the targets' Passive Perception - 5.
From there, simply follow the normal rules for surprise. What they can do when the round begins can vary - maybe they don't have enough movement to get close enough to do anything, maybe they're forced to dash, maybe they have to use ranged attacks. There is no "Surprise Round" in 5E, as in previous editions.
Advanced
Because the rear guard is also ignorant of when the advance team actually strikes, it may be desirable (or at least somewhat more realistic) to consider them as three "sides" in the engagement, despite the fact that two are naturally allied.
Compare the rear guard's Passive Perception (you may consider giving them +5 for advantage because they know something is coming, just not exactly when) to the advance team's Stealth rolls, too, then follow the normal rules for surprise. This makes it possible for the rear guard to be surprised during the first round, making them a bit slow to engage because they have to wait for the sounds of conflict to know when to move in.
Alternatively, another method to reflect the rear guard's imperfect information would be to impose disadvantage on their Initiative checks. This could be interpreted or explained as them waiting until they're absolutely certain they heard the fight begin before moving in.
Advantage & Disadvantage
Advantage and disadvantage are among the most powerful tools at the DMs disposal for rewarding player thoughtfulness. They exist as a quick way to grant a benefit or drawback if the situation is good for a given participant. It has an almost minimal impact on speed of play, but can make players feel like their non-mechanical ideas have impact. What the exact advantage is doesn't matter. What the exact disadvantage is doesn't matter. If they have a big enough one to matter, apply it.
Best Answer
The initiative rolls can be interpreted as a metaphor for other circumstances. So if the orc rolls high, imagine that as him turning his head ina lucky timeframe, just to see the attacker in the corner of his eye. But he is still surprised, so he does not get to act first.
To answer: the action that causes the surprise does count as a part of the round. The initiative is rolled when it is obvious that a combat should start, i.e. when one of the players or the DM decides to attack. Whether you roll it before or after determining surprise is of no consequence. But it is definitely rolled before the first attack (the back-stab in your case).
See e.g. this question. By the rules, it would go like this:
If Renee "goes" first: Renee uses her round to abuse the poor orc. Then the orc's round comes up, he cannot use any action or move on this turn and he couldn't have used any reactions up to the end of it (so for example if Renee moved away after the stab, she would get no attack of oportunity). From now on, combat goes as usual, next turn is Renee's.
If the orc "goes" first: The orc cannot do much on his round, since he is surprised. So his turn ends. From now on he can take reactions and normal actions on his following turns. Renee goes next and she spends her turn with the backstab (if she wants to). If Renee then wanted to move away from the orc, the orc could use the attack of opportunity on him. From now on, the combat goes as usual.
How this affects assassinate: If Renee beats the orc's initiative roll, she gets both the advantage (since she takes his first turn before the orc) and the crit (since the orc is still surprised)
If the orc goes first, Renee gets neither, since the orc is no longer surprised by the time Renee gets to her turn.