Why does "orange" rhyme with (almost) nothing in English?

Firstly, orange does rhyme with a few words: there's the word 'sporange' in botany (and related words hypnosporange, macrosporange, and megasporange) whose American pronunciation rhymes with 'orange', there's a hill 'Blorenge' in Wales, and it has been claimed (perhaps humorously) that in some dialects, 'door-hinge' is pronounced to rhyme with 'orange'.

But this is not what you meant; you were asking about common English words. So the question is "What is so special about 'orange' that no other common words rhyme with it?"

Laura Wattenberg observes (bolding mine):

Here's a little pet peeve of mine: nothing rhymes with orange. You've heard that before, right? Orange is famous for its rhymelessness. There's even a comic strip called "Rhymes with Orange." Fine then, let me ask you something. What the heck rhymes with purple?

If you stop and think about it, you'll find that English is jam-packed with rhymeless common words. What rhymes with empty, or olive, or silver, or circle? You can even find plenty of one-syllable words like wolf, bulb, and beige. Yet orange somehow became notorious for its rhymelessness, with the curious result that people now assume its status is unique.

In fact, this notoriety of 'orange' is so unjustified that Wikipedia even has a long article called "List of English words without rhymes" — and it notes that the list is seriously incomplete (among words it includes are music, month, depth,…).

Mark Lieberman at the Language Log, in a post on Rhymes, tried a quick exercise classifying words into rhyme sets, and found that his script:

… revealed 50,344 rhyme equivalence classes (i.e. sets of rhyming words), of which 30,905 (61% of rhyme sets, 16% of words+pronunciations) are singletons.

In other words, 16% of words (about one in every six) have no rhymes at all! And among initial-stressed two-syllable words (like 'orange'), he found 26% of words had no rhymes at all.

Though he admits there are bugs in the definition of rhyme he used, and it needs more detailed study, the general answer to "What is special about orange?" stands clear: nothing is special at all. There are a great many words in English without rhymes, and for some reason 'orange' gets mentioned as if it's somehow unique. There's no reason to expect every word to have a rhyme, and it's unsurprising that many words don't — you don't have to look to whether a word violates the phonotactics of English!