What is my bicycle flange nut size?

Do you have access to a thread gauge and a more-accurate measurement tool like a vernier caliper or a micrometer? The "Caliper and pitch gauge" section of https://www.wikihow.com/Measure-Thread seems approachable.

Basically you measure the outside of the bolt's threads and aim for the nearest "standard" size.

Then use the thread gauge to line up the points with the valleys and read off how many Threads Per Inch (or TPI)

The nut's thread has to match the bolt's thread.

---

Note that the outside of the nut is not directly related to the thread specs. As long as it fits in any recess on your bike frame, then its okay.

Another option is to carefully hand-file the daggy bits off your original nut, and create six flats.

Finally, stop using an adjustable spanner on this. You should use a ring spanner, or change out the whole bolt to a QR instead.

The thread pitch looks too coarse to be metric. Since a 3/8-16 (3/8 inch diameter with 16 threads per inch) was too large I would speculate its' a 5/16-18 (5/16 inch diameter with 18 threads per inch). The standard wrench used with this size nut is 1/2 inch. This is a common size on older American made bikes.

FWIW, I'm guessing your clamp looks like this, more or less:

Or is it like this: