My PS4 thinks I'm moving the joystick up when I don't touch it at all. How do I fix it?

Once you use the controller for a while, that's expected. There's no good way to mechanically repair it. You can try to set the deadzone in the game options.

This is a hardware problem called the "stick drift", where the analog potentiometers on the joysticks themselves start failing over time. This is caused due to the use of very low quality ALPS Potentiometers by Sony.

Your only solution is to go to a repair shop and request the swap of the entire joystick or just the analog potentiometers.

I had sometimes that problems on DualShock 3. Everything backed to normal after blowing at angle few times on the analogs.