How small can a flame get?

Fire has to be hot enough to burn. The smaller the flame, the cooler the flame, until it gets so cool that it doesn't burn.

Smaller flames are cooler because they have more surface area in proportion to their volume, so they radiate away heat faster.

Edit: When a fire goes out when it still has fuel and oxygen (assuming it wasn't smothered), it's because it was too small to be hot enough. For instance, the OP's candle flame that went down to a millimeter and then out, did so because the minimum size of a wax fire that's hot enough to sustain the reaction, is larger than the fire he had.

The minimum self-sustaining fire size will depend on the amount of oxygen and the geometry and type of fuel, because those factors affect the rate of heat production of the fire. If you are familiar with starting a campfire, the minimum size of a tinder fire is quite small because tinder has a large surface area for combustion, giving it a high rate of heat production, capable of keeping the tinder above the ignition point despite the high radiative heat loss for a small fire. The minimum size of a log fire is much larger because of the lower surface area, so the log fire has less heat production in proportion to its size. This is why you can't just put a match to the log, and it's also why using a small amount of tinder to ignite one end of the log is likely to fail as soon as you're out of tinder. You need to first use a tinder fire to heat a large section of the log above the ignition point, or several logs close together, before the log fire will be large enough to be hot enough to be self sustaining.

Combustion is... complicated. Essentially what is going on in flame is that you have molecules of fuel and oxidizer that mix and start to bounce off each other. If the molecules are moving fast enough (meaning they have enough energy, which we measure as temperature), then when they collide with each other, they start to make the fuel and oxidizer fall apart into other molecules.

Depending on which molecules collide and the energies involved, when things start to fall apart they are moving to lower energy states and the energy that was stored in the chemical bonds gets released as heat (and radiation in the form of light, which may be invisible). If it is happening often enough, the heat raises the temperature (adds energy) of the molecules around it and the process starts to run away. This is how you get a stable flame.

So this means there's at least a fundamental limit to the thickness of a flame -- you couldn't have a flame at lengths smaller than the distance molecules travel before they collide. This distance is called the mean free path, but frankly it's not a useful limit because flames cannot exist on the scale of the mean free path for other reasons.

For a flame to exist and be stable (i.e. not just a spark or something that goes away quickly), the rate of heat release has to be in balance with the rate of heat losses. If heat release exceeds heat losses, the flame will get bigger. If heat release is less than heat losses, the flame will run out of energy.

All of this means it is difficult, if not impossible, to put a general limit on the smallest possible flame. It will depend on the fuel source and how much oxidizer is present (different fuel+oxidizer combinations need different energies to start releasing heat), what the flow around the flame is like (how fast heat is carried away), how much energy the mixture has (higher temperature means more collisions that can break things apart), and how far the molecules need to move before they collide (how dense the mixture is).

The only definitive thing we can say is that the flame needs to be thicker than the mean free path, but anything more precise would require getting specific about the setup.

---

For a candle flame, we're looking at what is called a diffusion flame. The fuel (wax) is on one side and it has to vaporize and diffuse/mix with the oxygen in the air before it can properly burn. This is pretty hard. An overview lecture on diffusion flames is available, but it's actually not that easy to define a thickness for diffusion flames.

Suffice to say that the flames can be arbitrarily small, at least for sizes greater than several times the mean free path, provided the heat release is in balance with the heat losses. To be more specific would require a lot more details of the setup.