Why isn't every magnetic neutron scattering peak a nuclear scattering peak as well?

Neutron diffraction is a well-established technique for determining the magnetic unit cell of magnetic materials. The idea is that nuclear scattering gives you peaks that correspond to the crystal lattice (according to the Laue or Bragg scattering condition), but in addition you get peaks for the magnetic lattice which disappear at temperatures above $T_C$.

What I don’t understand is why you don’t get nuclear scattering peaks at the same spectral positions as the magnetic ones: After all, the scattering centers are in the same places (localized at atom positions) and my expectation would be that you still get the same peaks without magnetic order, perhaps just less pronounced.

According to basic Bragg diffraction, you get spherical waves around each scattering center and the only two conditions for getting a peak are that

1. these interfere constructively at long distances for the given angle and lattice spacing and
2. that the structure factor doesn’t become zero.

If this same process happens for nuclear and magnetic scattering centers, I don’t understand how these two can "split up" to give two entirely independent spectra. Is there a fundamental difference between neutrons scattered by magnetic and nuclear centers, respectively, which means they can only interfere with their own kind? Am I misunderstanding something very basic about scattering theory?