Why does the Stern–Gerlach quantum spin experiment conflict with classical mechanics?

Suppose you shot a large number of small classical magnetic dipoles with magnetic moment $vec{mu}$ through the field. Imagine the dipoles to be small enough that they could be treated as the particles of an ideal gas, and they are "boiled" out of some source into the magnetic field.

We would then expect each of the particle's components of velocity to be randomly distributed according to the Maxwell velocity distribution - because that is the classical result for an ideal gas. So the alignment of their magnetic moments would also start out random.

The dipoles would experience both a force and a torque from the magnetic field. The torque would cause them to rotate, and the force would, as you said, tend to line them up with the magnetic field, until they are in a minimum energy state. This alignment would take some time however, and, since they started out with random velocity and random orientation of their magnetic moment vector to the field, their final velocities once aligned would show some variation.

The key, though, that makes the motion of the particles vary AFTER they are aligned, is the nonuniform magnetic field. Suppose the field is in the z direction, and varies with z.

The particles are in a minimum potential energy state once aligned, with potential energy

$E=-vec{mu} cdotvec{B} = -mu B $

But the magnetic field $B(z)$ varies with z, so the dipole still experiences a force

$dfrac{partial E}{partial z} = F(z) = mudfrac {partial B(z)}{partial z}$

So the classical dipoles, with randomly distributed magnetic moment orientations and velocities at start, would drift in varying directions, hit various positions on the detector.

But if the magnetic dipoles were somehow constrained to be on only two possible initial directions, you would expect to see a concentration of hits on two locations of the detector, and nothing anywhere else. They would start out with only two orientations with respect to the field and end up being deflected into only two concentrations on the detector. They'd have only two end states of "lining up" with the magnetic field, and then drift apart due to the nonuniform magnetic field.

So the Stern Gerlach experiment is evidence that the magnetic moment of electrons in atoms, and thus electron spin, is quantized, because the results resemble the second case above, not the first. The initial direction of the magnetic moment of the electron is limited by quantization of spin.

The key realization is that the particles have an angular momentum $vec{L}$ aligned with their magnetic moment $vec{mu}$. Specifically, $vec{mu} = gamma vec{L}$, where $gamma$ is the gyromagnetic ratio.

So if such a particle enters a magnetic field, and we try to treat it classically, we would expect it to experience a torque, which will change its angular momentum: $$ vec{tau} = frac{dvec{L}}{dt} = vec{mu} times vec{B} = gamma vec{L} times vec{B} = -vec{Omega} times vec{L}, $$ where $vec{Omega} = gamma vec{B}$.

This can be recognized as the equation for gyroscopic precession. This implies that classically, the angular momentum (and therefore the magnetic moment of the particle) will not "flip" over to point along $vec{B}$. Instead, it will precess about the axis of $vec{B}$; and importantly, the component of $vec{mu}$ along $vec{B}$ will remain constant at all times.

So if we fired a bunch of classical particles into a Stern-Gerlach device with a magnetic field along the $z$-direction, we would expect them to be sorted according to their $B_z$ component. Classically, this quantity is entirely continuous, and so we would expect a continuous spread of impacts along the screen. Of course, this is not what we actually observe.