As a personal opinion, I'll suggest that there is no tidy single word term for gender specific words. In fact "gender-specific" or non-gender-specific or similar are probably as good as you can get.
No noun is immune to gender differentiation, or removal of gender differentiation. If somebody takes a term that is usually asexual and produces two new gender based variants that are recognisable, it will probably not survive, but there is no reason why an especially apposite creation may not become part of the language. Some words may have a general and eg female version but no distinctly male equivalent. eg the (contrived for the purposes of this illustration) term "pilotess" would be immediately understandable. I cannot think of a distinct male equivalent. [For female-er-ising, addition of "-ess" works in many cases ! :-) ].
Such choices either way, as have occurred in 'recent times', are liable to have been driven by the desire on the one hand to use gender inclusive language, and on the other hand to use terminology which makes a point about discrimination or differentiation when it is used.
The term dancer is indeed gender inclusive, but "ballerina" exists as a term which overwhelmingly suggests a female protagonist. The term "male ballerina" gives 14,000 Google hits - but most seem to be asking what the correct term is (Some suggest "Cavalier"). Some sites such as this one are so bold as to use the term directly , but still manage to revert to the occasional "male ballet dancers" indicating that the usage is unusual.
The suffix " ...ina" tends to suggest either 'small' or 'female' but this is not necessarily so in all cases.
For added fun, consider the gender inclusive / male only / female only versions of: Waiter, Host, Bellboy, Pointsman (cars), Point-man (guns), Aviator, Dominatrix, Seamstress, Druid, Governor, Best man, Minx, Cougar, Priest, Nun.
Many need extra discussion to explain variants. eg "Dominator" may be the proper male version of Dominatrix but loses a certain something. Priest may become priestess, but not always. Seamstress seems to have no equal. etc
Yes, it can be used for either gender. A Smart Alec is a neutral noun - there is no gender attached or implied.
Additionally, as @TrevorD commented, Alec, Alex, and other similar contractions are used for female names as well as for male names.
Similarly, while the root of 'Billy No-Mates' is male, it can be used to describe an unpopular man or woman.
Best Answer
There really is no general rule. Language evolves, and the evolution is primarily influenced by the people using the word, and different communities have different ways of thinking, so the “unisex” solutions turn out to be different for different words.
There is a critical distinction to be drawn here between at least three kinds of gender-neutral language.
One is grammatical neutrality: this is easy in English but hard in many languages. For example, moon has no grammatical gender in English, but is feminine in French (la lune) and masculine in German (der Mond). Nevertheless, a few words in English sometimes take particular gender pronouns: earth, moon, and nature, for example, certain moral qualities (such as wisdom and justice), and certain forms of transportation (such as ships and automobiles), are sometimes feminine and take the pronoun she. You might describe this as personification. Another term for it, according to the Wikipedia article “Gender in English”, is covert gender.
A second kind is etymological neutrality: language that contains no possible ambiguity, because it avoids root words that could be mistaken to mean only people who identify as a particular gender. In English, we rely more and more on such terms: attendant, parenthood, letter carrier, not stewardess, motherhood, fatherhood, postman.
A third kind is connotative neutrality: language having grammatical or etymological roots in a gender, but nevertheless used and understood to connote nothing about gender: manslaughter, freshman class, maiden voyage, master key, fraternal twins, lumberjack, matriculate.
In the fishing industry, the gender-neutral term actually used most is fisherman, plural fishermen. The term is neutral in the first and third sense, but not the second. It is grammatically neutral, neutral in connotation, but not neutral in etymology, much like freshman and lumberjack.
There was a campaign in Canada to adopt the word fisher, but the women in the profession largely refused to have anything to do with it:
It is certainly possible that the preference for fisherman as the gender-neutral term will diminish as time passes. Over the past thirty years, the use of many such words has become less common. As of this writing, fisher has not yet caught on in occupational or popular usage. Some academics and governmental agencies now use the term fisher, and this may eventually influence the public. See for example the conclusion of the recent Australian Broadcasting Corporation article, “Is ‘fishermen’ a sexist and exclusionary term?” But only time will tell whether fisher really will catch on.