British slang has lots of phrases for this, including 'fancy woman' and 'bit on the side', but I don't know if they'd be understood outside Britain.
'Paramour' is an archaic word. I wouldn't use it at all unless I was writing a fantasy novel. The same goes for 'concubine'.
'Mistress' doesn't have the connotations you describe to me, and is the word I would use if it wasn't already clear that the man was married. If it was, I would probably say 'girlfriend'.
A term that could be used for either gender is 'lover'.
"Call me when you are back" would be more natural during speech.
"Call me once back" looks like something that would be written in an email, by someone who is being terse and writing as few words as possible. Which is fine if that is your style.
In speech you would say "Call me once you're back."
Best Answer
In modern English "You" is both singular and plural but it always takes a verb form that originally marked the word as plural. This happens because of the way "to be" conjugates in modern English.
Examples: You are, you have, you weren't, and so on... They may all mean a single object or multiple objects.
In Early Modern English there was "thou" for the singular version of "you" and "ye" for the plural version of "you" but nowadays you will hear it mostly never.
If you wish to use "you" to speak about every single person of a group, use Each one of you, Every single one of you:
In English "you" can stand to refer to an unspecified person and is primarily used as a colloquial or less formal substitute for "one".