Why can't you just set the altimeter to field elevation?

Because air pressure is not constant over wide areas. Weather provides us with high and low pressure areas, and you may not know the pressure at your destination when departing. Indeed, it may have changed en route. OTOH, barring a major earthquake, the field elevation is not going to change significantly.

In point of fact, I DO almost always set the altimiter to field elevation when departing, mainly because a lot of my flying is from smaller fields that don't even have Unicom. If I'm at a larger field, I might cross-check with the reported pressure, just to make sure things are working correctly.

Good question. The reason is things fail.

If you set the altimeter to show field elevation, but the altimeter is faulty, you could end up flying differently from the rest, leading to loss of separation or worse.

But if you set the QNH, and then the reading is within 75 feet of the field elevation, you're good to go.

3. Altimeter Setting Procedures

3.1.3 If the difference from the known field elevation and the altitude read from the altimeter is plus or minus 75 feet or greater, the accuracy of the altimeter is questionable and the problem should be referred to an appropriately rated repair station for evaluation and possible correction. (USA AIP ENR 1.7)