Is there a word for the inability to hear a distinction between certain vowel sounds?

The first term that comes to mind is "tone deafness".

I don't know if this is the correct term for the inability you describe. Wikipedia says "Tone deafness is the lack of relative pitch, or the inability to distinguish between musical notes that is not due to the lack of musical training or education." The article in question does not mention hearing "the distinction between certain sets of vowel sounds."

I think we can call it "phonological unawareness". Or it can be related to the different levels of phonological awareness:

Phonological awareness refers to an individual's awareness of the phonological structure, or sound structure, of spoken words.

It is related with listening skills as well:

• Alertness: Awareness and localization of sounds

• Discrimination: Recognize same/different sounds

• Memory: Recollection of sounds and sound patterns

• Sequencing: Identify order of what was heard

• Figure-ground: Isolate one sound from background of other sounds

• Perception: Comprehension of sounds heard

Phonics deals with phonological awareness:

Phonics is a method for teaching reading and writing the English language by developing learners' phonemic awareness—the ability to hear, identify, and manipulate phonemes—in order to teach the correspondence between these sounds and the spelling patterns (graphemes) that represent them.

That is to say, it is usually mentioned regarding to teaching the language to the children or neurolingustic researches about bilinguals or non-native speakers.

Another source that mentions the term:

Accurate word learning requires identification of the sounds and letters in the word. Without such clarity, meanings are harder to learn; build, built, and bill differ only by one phoneme, as do bruise and breeze, and goal and gold. One of my fifth graders, years ago, was sure for weeks that the Gold Rush had something to do with soccer ("goal rush"), a semantic confusion directly tied to phonological unawareness.

---

And there is more to add to this topic. Phonological history of the English language.

Now that you mention it, we should talk about Mary–marry–merry merger also:

One of the best-known pre-rhotic mergers is known as the Mary–marry–merry merger, which consists of the mergers before intervocalic /r/ of /æ/ and /ɛ/ with historical /eɪ/. This merger is quite widespread in North America.

Another useful question: How are 'marry', 'merry', and 'Mary' pronounced differently?

When in college I wrote one paper 17 times, my teacher said I have sound deficit syndrome. Also when reading words, I know what they mean, I just can’t pronounce them correctly, especially medical terms. Background noise always bothers me. I can hear most anything when quiet, if TV on then it’s hard for me to hear.!
What would you say, I am 60 + now still same problem.