Why are infrared images not great?

It is often the number of pixels in the sensor that is limiting the quality.

The sensors are often made of a thermoelectric material. When it absorbs radiation, the temperature of the pixel goes up and this induces a voltage (like in piezoelectrics), which is then read out. It is probably useless to make small pixels because of thermal conductivity.

There is not much incentive to produce better optics. Good IR optics will be very expensive because of the small series.

It is certainly possible to use IR technology to capture detailed images, but such equipment is very expensive. So you won't see it much.

Very near IR can be handled with CCD sensors just like optical. I'm pretty sure you've seen some "night" images where targets are imaged via reflected IR. These images are (except for the usual monochrome) very similar to other optical images.

Once you get away from near IR (like the common thermal/FLIR images), the CCD sensors don't work any longer. All the decades of work on optical CCD sensors doesn't help you and you have to develop specialized items. The energy isn't sufficient to drive electronic transitions in the same way. The cheapest camera sensors (what you get on a <$1000 device) operate thermally instead. They don't need cooling, but the sensor is tremendously slower than way CCDs operate. Increasing the size of the sensor pixel reduces the noise. Add in some active cooling and some of these problems go away, but now you're paying for a cooling system....

You can do electronic transitions with the IR spectrum, but it requires new materials and new techniques. You can't pick those up in a commodity market for $20. Different technologies Wikipedia IR Sensors

Another cost problem is restrictions in the US market. You can build an optical camera and sell it anywhere. But IR sensors are ITAR controlled. If you make a hi-res thermal camera, your costs for selling it go up if you have to deal with lots more paperwork. Export Restrictions

So I'd say that the image quality for thermal IR is a combination of different aspects of your #1 and #3. High resolution FLIR images are possible, but there are very real technology and market forces that make such cameras expensive compared to the camera on a smartphone.

Google 'people infrared images' and what comes up are red/green/blue blobs of human outlines where the larger details are present. <...> I'm just curious about what is stopping IR photography from being as detailed as visible photos.

In fact, IR photos, if we take them in the same resolution as the visible ones, are as detailed as they should be. I suppose what you're thinking of as "lacking details" is not related to resolution though.

Let's consider an IR image from this article:

If you look up into the sky, you'll notice the wires and their support structures in quite good detail. Similarly, the reflections from the train are quite detailed.

OTOH, we don't see much texture on the wall or the floor, where in visible range we would expect some roughness. But this is not due to limitations of the cameras: it's what the radiation field actually is.

Remember that when you image far IR radiation, it's mostly emission from the objects themselves, rather than—as would be normal for visible—scattered ambient light or sunlight. It's more correct to compare far IR images to the visible-range images of glowing metal or glass.

But, unlike these hot things, the typical IR-imaged objects at room/street temperatures have much smaller temperature gradients due to being roughly in equilibrium with the surrounding air. Thus, even though glowing piece of metal may look "textured" like in the image below, it's only because of its highly non-uniform surface temperature.

^{(image source)}

Such a non-uniformly-hot object would also look "textured" in the IR, like here:

^{(image source)}

Conclusion: the reason why the typical IR images aren't too detailed is that there isn't much detail in the actual radiation pattern to be captured.