Firstly, orange does rhyme with a few words: there's the word 'sporange' in botany (and related words hypnosporange, macrosporange, and megasporange) whose American pronunciation rhymes with 'orange', there's a hill 'Blorenge' in Wales, and it has been claimed (perhaps humorously) that in some dialects, 'door-hinge' is pronounced to rhyme with 'orange'.
But this is not what you meant; you were asking about common English words. So the question is "What is so special about 'orange' that no other common words rhyme with it?"
Laura Wattenberg observes (bolding mine):
Here's a little pet peeve of mine: nothing rhymes with orange. You've heard that before, right? Orange is famous for its rhymelessness. There's even a comic strip called "Rhymes with Orange." Fine then, let me ask you something. What the heck rhymes with purple?
If you stop and think about it, you'll find that English is jam-packed with rhymeless common words. What rhymes with empty, or olive, or silver, or circle? You can even find plenty of one-syllable words like wolf, bulb, and beige. Yet orange somehow became notorious for its rhymelessness, with the curious result that people now assume its status is unique.
In fact, this notoriety of 'orange' is so unjustified that Wikipedia even has a long article called "List of English words without rhymes" — and it notes that the list is seriously incomplete (among words it includes are music, month, depth,…).
Mark Lieberman at the Language Log, in a post on Rhymes, tried a quick exercise classifying words into rhyme sets, and found that his script:
… revealed 50,344 rhyme equivalence classes (i.e. sets of rhyming words), of which 30,905 (61% of rhyme sets, 16% of words+pronunciations) are singletons.
In other words, 16% of words (about one in every six) have no rhymes at all! And among initial-stressed two-syllable words (like 'orange'), he found 26% of words had no rhymes at all.
Though he admits there are bugs in the definition of rhyme he used, and it needs more detailed study, the general answer to "What is special about orange?" stands clear: nothing is special at all. There are a great many words in English without rhymes, and for some reason 'orange' gets mentioned as if it's somehow unique. There's no reason to expect every word to have a rhyme, and it's unsurprising that many words don't — you don't have to look to whether a word violates the phonotactics of English!
I admire your will to contrive history to turn Old Mother Hubbard into a master work of literature. Unfortunately in this case, I don't think there's terribly much evidence for these words rhyming (or at least, not for a majority of speakers). If you look at dictionaries from the 19th century, the words "bone" and "none" are squarely transcribed with different vowels, as are "coffin" and "laughing". (There's a chance that, once upon a time, more speakers did pronounce the vowel of "laugh", and indeed other words such as "dance", similar to that of "coffin", but sadly not at the time Old Mother Hubbard was written.)
If you want to be euphemistically kind to Old Mother Hubbard, then you could call it a "visual rhyme". An alternative theory is that Old Mother Hubbard is actually not a master work of literature. Shock horror.
Best Answer
The names come from French, where (from French Wikipédia)
That is, a rhyme was called feminine if the words ended with a mute e. Back when the rules for French poetry were formulated, these e's were pronounced, but unstressed, and one name for them was feminine e's. Mute e's are still pronounced when reading poetry and when singing, although not in normal speech.
Why were they called feminine e's? It probably didn't have anything to do with mute e's being weaker or more girly-sounding in any sense; they were called feminine e's because, to turn male adjectives and some male nouns into female ones, you added a mute e. For example, in French, a big black cat is:
This rule in French applies only for adjectives and some nouns which have male/female versions (e.g., chanteur, chanteuse); there are quite a few masculine nouns that end in mute e's and feminine nouns that don't.
You can see that the name "feminine rhyme" originated in French by looking at the reference (from London, 1764) T Romano gives in his answer, where a feminine rhyme is defined as one ending in an e-mute. In English, words (e.g., state and gait) rhyme whether or not they end in a silent e.