Why is subject-verb agreement violated in the following dialogue?

Hello 🙂 I was watching Big Bang Theory and there’s a scene where Sheldon’s brother says like this; I know life has been hard for you… that don’t mean it’s been easy for the rest of us.

When I googled "that don’t" certain verbs seemed to come after it like "that don’t mean," "that don’t impress," or "that don’t sound." Maybe there’s some nuance that this subject-verb agreement violation implies but I couldn’t get any information. I’d really appreciate it if you could help me understand it. Thanks so much 🙂