Generally, I do not allow surge of fortune to work with vorpal and similar, and in general do not allow any effect that replaces a roll with a fixed number to trigger effects that occur on certain rolls. But my players also know this before they build characters around it. Barring that...
A dragon is a high-level spellcaster. It should fight like a high-level spellcaster. Red dragons somewhat less so than others, but an ancient red dragon is smarter than any real-life human being has ever been, is centuries old, and has been a force of greed and malice, death and destruction, for ages; one does not get that old by charging in blindly to face unknown foes.
One, a dragon has to assume that being hit equals death. It usually does if a foe knows what he’s doing (shivering touch is traditional for one-shotting dragons, to the point that in-character basically all dragons in my campaigns know it and don’t go into combat without prismatic scales as protection against it). The fact that the one-shot kill available here is a vorpal weapon used with surge of fortune is not necessarily important here.
Thus, I expect that any dragon worth his salt knows wings of cover and reserves several spell slots with which to cast it. This spell from Races of the Dragon is an immediate-action “negate one attack” spell; as long as he’t got spell slots, one attack per round cannot hit him. It’s also dragon-themed and sorcerer-only, both very appropriate for a dragon.
A thrown dagger cannot be thrown more than 50 ft. (barring Far Shot or whatever). A dragon has no reason to let anyone get that close to him; his movespeed is gigantic, and spells allow him to attack at range. Fly-by Attack is an excellent feat that allows any standard action (like casting a spell) to occur anywhere in the middle of movement: that allows the dragon to move in and out of range while sniping with spells. If it happens that the rogue does get his 50-ft. throw, wings of cover.
Starting with greater dispel magic is a very-solid play for any spellcaster. If that eliminates the rogue’s surge of fortune or suppresses his vorpal weapon, the threat is nullified. Of course, the dragon won’t assume that it did; he’ll maintain range and again, reserve wings of cover.
From there, effects that immobilize, separate, and shut down the party members are very potent. Stick a fighter or a rogue in a solid fog, and he won’t be going anywhere for a while. Make it an incendiary cloud if you want to play up the red-ness. Using forcecage is pricey, and dragons, particularly red ones, are notoriously loathe to give up any treasure, but forcewall isn’t and is still very effective. Offensive uses of teleport effects, well timed, can be brutal. An ancient red dragon could make for a decent quasi-mailman, too, with orb of fire and Searing Spell.
Defensively, mind blank is a good idea to just have; an ancient dragon can probably afford it in item form, or an ancient red dragon could simply know it as his 8th-level spell (i.e. can’t have it and incendiary cloud). Death ward is also a gimme, considering the red dragon’s access to cleric spells. Stuff like mirror image, displacement, and blink serve to make attacks that get through wings of cover still have a high rate of failure, even assuming a natural-20. Since getting locked down is dangerous (his huge mobility is a major advantage, and necessary to offset their action-advantage), freedom of movement is great.
Finally, don’t forget the arcanist stand-byes: nerveskitter means the dragon probably goes first, celerity means he definitely goes first, and contingency means one thing he isn’t able to proactively protect against is nullified anyway (“If I have already used wings of cover and an effect targets me rather than one of the mirror images, greater teleport me [somewhere safe/out of the line of fire/whatever].”)
Frankly, with all these spells available to him, the ancient red dragon is going to be monstrous. He’ll be almost impossible to keep in range, almost impossible to hit even if you do, have several options for negating effects, and so on. He’s right around the threshold where a sufficiently-paranoid spellcaster cannot be realistically attacked, and one presumes an ancient dragon will be sufficiently paranoid.
No, the creature still loses its head
The effect of the vorpal sword is explicit:
...and roll a 20 on the attack roll, you cut off one of the creature's heads.
This wording in the item description implies that the beheading effect is completely distinct from the effect of a critical hit. Reducing the result of the attack from a critical hit to a regular hit doesn't change the fact that the attack rolled a natural 20. So, adamantine armor won't stop the sword from beheading you.
Other weapons with similar wording behave the same
Items like the sword of sharpness and mace of smiting have similar wording. As such, they work identically. If you roll a natural 20 on your attack roll, then you apply the extra effects from the item, regardless of whether or not the result is a critical hit.
However, effects that apply "when you score a critical hit" are negated
For example: The nine lives stealer description states:
If you score a critical hit against a creature that has fewer than 100 hit points, it must succeed on a DC 15 Constitution saving throw...
In this case, when the attack scores a critical hit against you, the critical hit becomes a mundane hit. So, the extra effect from this kind of weapon doesn't apply since the attack didn't score a critical hit.
Best Answer
Armor of Invulnerability specifically only gives resistance/immunity to damage form non-magical weapons. A vorpal sword is a magic weapon. Therefore any damage it does, including that from the vorpal effect, is from a magic weapon and thus from a magical attack.
As such the resistance/immunity from the armor does not apply.