Putting car into any gear for the first try isn't smooth

If this applies to every gear, I would guess something in the gear selection linkage is worn.

If it applies to some gears but not others, more likely something in the gear selection linkage needs adjusting.

Once you have selected a gear, putting the stick back into neutral is not misaligning whatever made it hard to select the gear the first time.

Try deselecting a gear, then waggle the stick from side to side in neutral and reselect the same gear. If that makes it less smooth, it tends to confirm the "worn linkage" theory.

Either way, it shouldn't be hard to fix it. You don't need to pull the gearbox (or something equally expensive to do) just to adjust the linkage or replace a worn part.