How can a bad lower ball joint cause the car to suddenly veer off course?

My car, a 2002 Ford Escape, suddenly veers off course when I’m driving. When you take the front passenger side tire off there is a lot of play around the stearing knuckle. I don’t see any obvious problems with the tie rod end or lower ball joint but I’ve heard those are the two things to check.

I have a suspicion it is the lower ball joint but like I said I don’t see anything wrong with it. I don’t quite see why that being damaged would allow the whole stearing knuckle to have as much play as it does. Because it never does anything outside of the normal operating parameters of the lower ball joint when I move it with my hands.