In response to Jacinto's comment, I searched again and found a reference to a man being a wallflower, in which it was not so gender-specific, although in the context it is already known that they are talking about a man. In 1913, Corra Harris wrote In Search of a Husband. On pages 47-48 I found the following passage of conversation between two women:
"Still it was queer, not to introduce him," I insisted
"Not queer, merely impudent. But he had the best of it. The difference between being a wallflower and an indifferent celebrity upon such as occasion is accomplished by turning your back to the company. Wallflowers sit or stand face forward, confessing defeat. He turned his back on us and put us out of countenance."
I've located a couple of other early 20th century references to male wallflowers, which are much more gender-specific.
From 1910, in At the home plate, by Albertus T. Dudley, page 123:
"I wonder what kind of a kid Crusty was when he was at school!" he thought, as he staggered back to his seat. "I'll bet he was a wallflower. It's queer that he should have a brother who can play football."
From March 1918, in The Recruit. A pictorial naval magazine. v.4 no.3 at the end of the first paragraph of page 24, from a short story called "The Wall Flower":
He was a wallflower
When there was a happy crowd grouped enthusiastically about the piano in one of the girls' homes, singing, laughing, and at ease, where was he? Sticking out as solemn and aloof as a sore thumb, that's where.
Further back in 1884/5, in The Freemason's repository. v.14 (following a list of questions regarding active participation in the Freemasonry), on page 133:
Is it not a lamentable fact that in the great majority of instances these questions would have to be answered in the negative? And now, by virtue of his office, he is entitles to a seat in the Grand Lodge. Is it any wonder he is a wall-flower there. Is it strange that the business of the Craft falls into the hands of a few men, who at the sessions of the Grand Bodies are overwhelmed with work?
Best Answer
This extract from Grammarphobia, while tracing its origin and usage, comments that "handsome" has had many meanings since it entered the English language in the 15th century, but why "good-looking" is mainly used to refer to man is still unclear:
Ngram shows a clear usage preference of "handsome" for men from the beginning of the 19th century.