How to explain the long lifetime of Rydberg atoms with Fermi golden rule?

These orbitals have many orbital nodes ($n-1$ when $n$ is the orbital quantum number which can be a few hundred). So the radial wave functions are rapidly oscillating, and contributions to the overlap integral with the dipole operator tend to cancel.

Indeed the matrix element between an outer Rydberg state to a lower lying Rydberg state is much larger than the matrix element between the outer Rydberg state and the ground state. However, the decay rates depend also on the density of electromagnetic modes, which depends on the frequency of the transition. In particular, the decay rate $Gamma sim omega^3$, so transitions emitting higher frequency photons receive more weight from this term.

One note is that transitions from outer Rydberg states to other nearby Rydberg states, for which the energy gap is small, can have associated photons in the infrared that can actually show up in blackbody radiation. In fact, blackbody radiation at room temperature can stimulate transitions between Rydberg states and can be the primary limit in Rydberg state lifetime, rather than spontaneous emission.

For a chart showing spontaneous emission rates and stimulated emission rates from the n=30 Rydberg level to other states, see Figure 1 of https://arxiv.org/abs/0810.0339.