How to fix joystick bumps

With my old pair of joy-cons (and my new ones, too), I have noticed that the joysticks have some sort of bumps on them that look a bit like this:

I believe that this is because the lid of the joy stick has lots of little edges that have to fit into smaller holes in the actual plastic part of the joy stick.

I have seen that there are fixes such as moving the lid around and using a blow dryer to make it better, but I want to avoid melting or breaking apart any part of my joy-stick.

Is there any way to remove these bumps with no or minimum damage?