[RPG] What happens when you non-lethally attack with a vorpal weapon but roll a 20 on the die

Yup. Roll 2d20, take the higher.

When you have either advantage or disadvantage, you roll a second d20 when you make the roll. Use the higher of the two rolls if you have advantage. (PHB p.7)

There's no "you may" or "you can" in there, you just do.

(Note that some sources of advantage are optional: if you choose not to "accept" advantage then you, of course, wouldn't roll two and take the higher. And it's not hard to give oneself disadvantage, opposing your advantage. But your question said that you have advantage, in which case there's no option on the dice.)

(Note also that the Lucky feat interacts with (dis)advantage in a counterintuitive way according to Sage Advice.)

It is usable as a weapon; additional effects depend on the specific magic item

The wielder either gets the effect of the magic weapon, or they do not. The answer depends on the nature of the magic weapon.

Case 1 – flat modifiers (+1/2/3/etc..): The wielder gets the effect

The "Identifying a Magic Item" section in the DMG (p. 136) has a few paragraphs describing interactions between a character and magic items that might clue them into their properties or how to use them:

Sometimes a magic item carries a clue to its properties. The command word to activate a ring might be etched in tiny letters inside it, or a feathered design might suggest that it’s a ring of feather falling.

Wearing or experimenting with an item can also offer hints about its properties. For example, if a character puts on a ring of jumping, you could say, “Your steps feel strangely springy.” Perhaps the character then jumps up and down to see what happens. You then say the character jumps unexpectedly high.

A straight bonus to attack and/or damage of a magic weapon is present regardless of whether the character is aware of the bonus or not.

In the case of a weapon with no activation required, the magical bonus is activated simply in the course of using the object as it would normally be used.

Following the guidance in the DMG, a GM could offer additional description to a hit such as, "The blade strikes true, and slashes through the hide of the beast with surprisingly little effort."

Case 2 – passive effects (e.g. mace of smiting) – The wielder gets the effect

Magic weapons with other passive effects (that don't require activation) can be adjudicated in the same fashion as the flat modifier.

In the case of the mace of smiting, a clue to the effect could be in a similar fashion to the boots of jumping example given in the DMG. The GM could add, "The mace connects with the golem with an unusually crisp ringing. The golem shudders as the strike seems to shake its entire body."

Case 3 – effects that require activation (e.g. dagger of venom): The wielder cannot use the activated effect

The character must have sorted out how to activate the item before the activated effect can be used. The passive effects that do not require activation would still be applicable immediately. The DMG lists some options of how to do that in the section on identifying magical items. Briefly, the options for figuring out how to activate the effects of a magic item are:

• The identify spell
• Focus on the item throughout a short rest
• Experimentation

Case 4 – items that require attunement: The wielder does not get the effect

In the case of a magic item that requires attunement, none of its magical properties operate until the character is attuned, as per the DMG/basic rules on attunement:

Some magic items require a creature to form a bond with them before their magical properties can be used. This bond is called attunement, [...]

Without becoming attuned to an item that requires attunement, a creature gains only its nonmagical benefits, unless its description states otherwise. For example, a magic shield that requires attunement provides the benefits of a normal shield to a creature not attuned to it, but none of its magical properties.