Learn English – Word for a “Male Mistress”

As a man, I still consider it derogatory and would not use it in any normal conversation. I will agree that it's quite mild as a term, not a major insult, but it's still derogatory.

It's originally a slang term, derived (if I'm not mistaken) from the older British slang of "bird" for a woman. (Thus, young woman = "chick".) It was almost invariably used between men, with the "bird" being a form of quarry to be hunted, and the term still has that connotation. It is definitely dismissive; the person referred to that way is not on an even status with the speaker.

As far as groups that refer to themselves with it, such as the cited "Dixie Chicks" and "Chicks with Picks", I consider them to have understood the connotations and intentionally used it ironically. I'll refer to the organization as they prefer, but personally I still wouldn't walk up to a random member of "Chicks with Picks" and say, "Hey chick, come over here." At least, if I did I wouldn't expect a positive reaction. :)

Paramour may be the word you're looking for although it can be used for either sex.

an illicit lover; a person with whom someone is having a romantic or sexual relationship and especially a secret or improper relationship.

Her husband found a love letter from her paramour. source