Rutherford gold foil experiment

Consider that to get diffraction effects you generally need the wavelength of the diffracting particle to be related to a length scale of importance in your sample. For the moment, take it to be something on the order of an inter-atomic spacing, roughly an Angstrom ($pu{mathring A}$).

Taking an alpha particle at $pu{3MeV}$ (the ones originally used were more on the order of $pu{5MeV}$, depending on the source used). The de Broglie wavelength is straightforward to calculate, and turns out to be $pu{8 times 10^{-15} m}$, or $pu{8 times 10^{-5} mathring A}$, 4+ orders of magnitude smaller than an inter-atomic spacing. So, one can rule out diffraction as a cause of energetic alpha particle scattering.

Now, plenty of folks do use neutron scattering to measure structure in materials. The trick is that they have to be very slow neutrons, or 'thermal' neutrons. Going through the de Broglie equation for a $pu{25 meV}$ neutron (millielectron Volt, or basically room temperature) you get about $pu{1.8 mathring A}$ - which is right there with inter-atomic spacing. Now you have diffraction from solids!

So, energetic alpha particles are not diffracting off of anything related to crystal structure. They are interacting with a strong $1/r^{2}$ central force to give the observed differential cross section for scattering. And that force is centered on something heavy so as to result in backscattering. This shows the existence of the nucleus. (Another effect can be real interactions between nuclear energy levels and the alpha particle, but that is getting in to nuclear physics, a topic for another day.)

As a follow up to some of the comments - alpha particles do indeed interact with electrons in the sample, but their mass is so small that standard kinematics shows that an alpha will not backscatter off of an electron (unless it is highly relativistic - an alpha has a rest mass of about $pu{3.6 GeV}$, an electron has a rest mass of about $pu{511 keV}$).

Reflect means to bounce back, as in a ball bouncing off a wall. The average density of an atom is very low, so the observed reflection was startling... like throwing many billiard balls at a mass of fluffy cotton candy and every now and then a ball comes bouncing back!

The explanation offered was that there must be some "hard", dense object inside that fluff... the nucleus. Electrons are effectively spread out in their orbitals, and tend to act individually (bound by electromagnetic forces). An alpha particle, massing $pu{4 times 10^{9} eV}$, could not be reflected by an electron massing ~$pu{5 times 10^{5} eV}$ -- that would be like bouncing a baseball off a ping-pong ball.

Since both alpha particle ($ce{He}$ nucleus) and gold nucleus are positively charged, and the gold nucleus is held together by powerful nuclear forces (so all its particles' masses are effectively combined to ~$pu{2 times 10^{11} eV}$), it explained the reflection. Since reflection occurred only rarely, it implied that the nucleus was a small, dense target.