Intuitive argument for why $Delta m=0$ for transitions with light polarised in $z$-direciton?

The selection rules have $Delta m =0$ for light polarised in the z-direciton trying to excite an atom. Mathematically it is clear that $langle n, l, m_l| z |n, l, m_l rangle = 0$ unless $Delta m=0$. Therefore I understand the mathematical origin of this fact, I’m just looking for the most intuitive physical description of why this is true.

I have thought of the fact that there are no $x$,$y$ terms in the electric dipole radiation term in the Hamiltonian means that $m_l$ must be conserved through these transitions, but again this seems more ‘mathematical’ than intuitive.

I guess maybe it could just be geometrically obvious that any direction that is distorted solely by its z-coordinate cannot have its direction with respect to the z-axis affected?