What size are the nuts holding the front wheel?

Could be 5/16" which is very close to M8 (7.9375mm vs. 8mm). Thread might be UNF 24 (24 threads per inch) or British Cycle Std. at 26 threads per inch. There are even some that are metric in terms of diameter but use BSC for thread spacing! If it's 5/16" at 26 threads per inch then an M8-1.00 nut may fit since 25.4 tpi is very close to 26 tpi and 7.9mm diameter is very close to 8mm.

Assuming that the thread on the axle that you are trying to fit a nut is metric, you need to be aware that with metric threads there are 3 different pitches on every size. So, in your case you need to know if the threads on your axle are 8mm course, 8mm fine or 8mm extra fine.

Also as already mentioned, your axle could be an imperial size not metric, for example 3/8" is a common size

It is a M8 1.0 nut. I found out the hard way because all the hardware store has was a M8 1.25, so I bought it and the threads were way too coarse

First try your LBS, take the wheel with you.

You might save yourself pain in the long run by swapping out the axle for a standard size thread.

Park Tool list a number of common sizes.

3/8 inch x 24 tpi Some solid axle bikes, including coaster brake
3/8 inch x 26 tpi Solid rear axle
10mm x 1mm Most quick release rear axles
10mm x 26 tpi Rear axle, quick release, Campganolo®

Note that the difference between 24 and 26 tpi is small and can be subtle, meaning that if you're using a spanner to do the nut up you can wreck the nut without really noticing that anything is wrong. The symptom is often that the nut strips before ever becoming tight. To avoid this put the nut on by hand until it is right over the axle. If it still spins freely it is the right thread pitch.

Less commonly you will find 12mm or 14mm axles, often on load bikes and Asian or African bikes, and some mountain bikes. But those are fairly obvious. Some older front axles were 9mm or even 8mm, and they bend easily. It's sometimes possible to replace a 9mm axle with a 10mm one, depending on the bearing cups you have available.

The below answer is copied from ( http://www.parktool.com/blog/repair-help/basic-thread-concepts )

Bicycle Industry Threads

The bicycle industry has a long history of using many different thread standards. Both factional and metric sizes are in use. Some threads are also used almost exclusively in the bicycle industry. Below is a table of some of the threads and their uses. This table is not intended to be complete and exhaustive. Always measure diameter and pitch when possible to determine threading.

Nominal Thread Size     Example of Bicycle Uses
2.2mm x 56 tpi  Common 2mm spoke threading
3mm x 0.5mm     Dropout adjustment screws, some derailleur hardware, accessory hardware
4mm x 0.7mm     Some derailleur limit screws (DIN standard)
4mm x 0.75mm    Common derailleur limit screw (JIS standard)
5mm x 0.8mm     Many uses on bicycles, including derailleur wire pinch bolts/nuts, disc rotor mounting bolts, fender and racks mounts, water bottle cage bolts, and others
6mm x 1mm   Many uses on bicycles, including brake caliper mounting bolts, brake pad bolts/nuts, some fender racks, some brake adjusting barrels
7mm x 1mm   Some handlebar binder bolts
5/16 inch x 24 tpi  Front hubs, solid axle, less expensive bikes
8mm x 1mm   Square-type crank bolts, front solid axle hubs, suspension system hardware
8mm x 1.25mm    Stem hardware, stud type crank nuts, suspension hardware
8mm x 0.75mm    Chainring bolt
9mm x 1mm   Front hubs, quick release, Asian manufacturer
9mm x 26 tpi    Front hubs, Campagnolo®
3/8 inch x 24 tpi   Some solid axle bike, including coaster brake
3/8 inch x 26 tpi   Solid rear axle
10mm x 1mm  Most quick release rear axles, derailleur mounting bolts, brake lever adjusting barrels
10mm x 26 tpi   Rear axle, quick release, Campganolo®
12mm x 1mm  Some spline crankset bolts
1/2 inch x 20 tpi   Pedal threads, one-piece cranks
9/16 inch x 20 tpi  Pedal threads- common three piece cranks
14mm x 1mm  Oversized frestyle axles
15mm x 1mm tpi  Crank bolt, Octalink® and ISIS Drive®
1-inch x 24 tpi     Threaded headsets, one-inch standard
1-1/8 inch x 26 tpi     Thread headset, 1-1/8 inch standard
1-1/4 inch x 26 tpi     Thread headset, 1-1/4 inch standard
1.37 inch x 24 tpi  Bottom brackets, ISO/English/BSC, and threaded freewheel hubs
1-3/8 inch x 26 tpi     Bottom brackets, older “Raleigh” three speeds
36mm x 24 tpi   “Italian” threaded bottom brackets

BSCy or BSC (British standard bicycle) it's a 3/8" diameter and 26 thread per inches. Thread are 65 degrees.. hard to find and expensive.. better off replacing the axle and nuts with a 3/8" 24 threads per inches.