Concrete vs crushed gravel for fence posts in clay soil?

Your reservations are very warranted. I've never had a problem in using just the soil I took out to put the post in. Even posts that warped severely after installation moved the fence, not the ground.

The good thing about clay is that water runs off and doesn't penetrate deeply at all. So, stay away from concrete, it retains moisture to the full depth and rots the post. Same goes for gravel, it just creates a pocket to receive water to then drain into the soil. Crown around your posts to force water away and keep fence panels/pickets 1/2" off the ground, you'll be set for decades.

In clay soil, the water just sits in the hole, especially where the water table is close to the surface. Gravel lets the water get right up against the post, which will rot it out fast. Using concrete will help the post to last longer, if you create a dome of concrete above soil level and make sure it is smooth and tight against the post. This will ensure that no water gets down into the post. Also, make sure the bottom of the post is not below the concrete level or the post will soak water up from the bottom (pour a little concrete in the hole, then set the post, then fill the rest of the way)

I always use exterior clear sealant where concrete meets post at base. This keeps water from getting between concrete and post. I also brush on a waterproofer on entire post that is above the concrete. I then spray the post a rubberized undercoater to about 4 or 5 inches above the base. Very effective in case any water or dirt contacts the post. I use pressure treated posts. Takes no time at all and a very economical way to keep posts lasting many years.