Why can't I get my caster wheels to engage their inserts?

I drilled a hole into 4 desk legs to add caster wheels. Added the metal inserts into which the casters will slide into. I can’t manage to push the wheels into the the inserts. For some reason even banging them with a hammer doesn’t move them much. Any ideas?