Why are there so few English words that begin with the letter X?

Your dictionary goes further than Johnson's, for which the entire chapter for X was thus:

X Is a letter, which, though found in Saxon words, begins no word in the English language.

And actually, it's not found in that many Saxon words. Saxon itself was one exception; Seaxe in Anglo-Saxon, as was the seax, the knife from which they took their name. (The Old High German equivalent was Sahsun though, the X wasn't shared with all their neighbours).

While the the Latin alphabet adapted (with the addition of Ƿ Þ & Ð and the promotion of Æ from digraph to letter in its own right) for English use had an X, and before that the Anglo Saxon Futhorc had ᛉ, the ancestor of the Futhorc, the Elder Futhark, had no such rune. [Rect: It had the rune ᛉ, but for a different sound].

So X it would seem was a bit of a novelty. It's also mainly used for a sound that cannot start a syllable in English. Notably, some English words that do start with X come from Greek words that do start with that sound (from Ξ rather than from Χ) get pronounced as do those beginning with Z. These words are also mostly relatively recent imports. (Though Χ is where X originates it has a sound that gets imported into English with the hard CH of Christ or is changed further).

There was a brief period of spelling words that begin with sh or sch with an X in some areas (xal as a spelling of shall), but it didn't catch on.

So, while it was in the English alphabet from the beginning, it wasn't in the alphabet before it, and it was used for a sound that English never uses in the start of a word.

This doesn't give that much of a why, since we can still ask why we don't have the same sounds as Greek, and why the Germanic languages didn't have them to begin with, but that goes beyond what I know on the matter.

I think it can be interesting for analogy to look at the influence of English upon Irish and Scottish Gaelic in relation to the word vote; In both cases in seeing the need to have such a word, and lacking a letter V, the Irish imported the letter and produced votáil, while the Scottish adapted their existing tagh (to choose or select). It could have gone differently (the Irish have togh related to tagh), people speaking either language may argue about whether it is better to import or adapt, but it went the way it did.

And here we're seeing from the outside what happened to English with such Greek words. It could have adopted Greek words that begin with Ξ more directly (learning how to pronounce the sound, instead of changing your xeric example among others) and earlier (so we'd have more such words), and more often, but it didn't.

(I'm sure there are other possible sources of X-words, but I don't know them).

And so, what X-starting words we have are more recent adoptions from Greek, recent coinages, and X-ray that use it as a symbol (it comes from the use of X for an unknown quality).

Edit: It's worth noting that we could easily do without X (a fact which makes it useful in languages only recently written in Latin script - you can press gang X into anything that doesn't fit the existing set of letters without much loss). Of course various alphabet and spelling reformers have said that of other letters, and that makes it no more likely to happen, but we actually did with connexion which while still sometimes spelt that way is more often found today as connection. Noah Webster argued for the latter "for the sake of regular analogy, I have inserted connection, as the derivative of the English connect, and would discard connexion." which in a way was the invention of a new word by deriving from connect which happened to have the same meaning and pronunciation as the existing connexion. (Connexion was not derived from connect though they did have their roots in derivatives - indeed we briefly had both connex as well as connect but it died out, compare the long coexistance of jail and gaol).

In a way therefore, it's as remarkable that we have as many words with X (though not at the beginning) as we do. Various words from the *laks Indo-European root for salmon, for example come in with an X where the source does not: We have lox though the original Yiddish is often transcribed laks. While the X in our gravlax isn't unheard of among the Scandanavian sources (gravad lax in Swedish, graflax in Icelandic) some others did fine without it (gravlaks in Norwegian, while the Danes use both that and also gravad laks). When it appears in Viking-originated placenames in Ireland and Scotland an X is used in the Anglicisation though it isn't in the Irish or Gaelic (Leixlip is Léim an Bhradáin in Irish, Laxdale is Lacasdal in Gaelic - happily moving away from the KS sound the Vikins used entirely).

In all it seems that while we would not go quite as far as Christopher Robin, who noted that the environs of Winnie the Pooh's house included "Big stones and rox", our instincts haven't been that far off his, at least until the 19th Century, when poor connexion began to get it in the neck.

So since so many of our neighbours, linguistically, make less use of it, it's as much of a curiosity that we do use X as often as we do.