Meaning of a phrase '... reciprocal of neutral density value required arranged on an inverted scale'

From earlier in the paper:

The intensity of the flash was controlled by Kodak neutral density filters in the path of the projection beam.

Neutral density filters are those that are intended to filter equally across the visible light spectrum. They're using them to control the light intensity (a lot more common back in the day when it wasn't as easy to control brightness electrically due to the light sources used, though I believe still commonly used in psychophysics experiments and certainly still used in microscopy where we still use some older types of light source). The "reciprocal of neutral density value required" is an effective luminance value, because "more neutral density (filter)" means "less light", so 1/(neutral density filter) is increasing as light is increasing. It's a unit of convenience since the experimenter knows the strength of their filters.

As far as the bars above the numbers, these are clearly to indicate negative numbers though I agree the notation is unusual, it may be a relic of an earlier day in typesetting. There is a more standard notation to indicate the first digit of a logarithm as negative with a bar (see Wikipedia; see also here where it is referred to as a "vinculum" but again only for some of the digits in a number rather than the whole thing) but that doesn't quite match the usage here.

are these log relative luminance values?

In the figure posted, these are log absolute luminance values, measured in units of log(foot-Lamberts). Negative logarithms indicate luminance values less than 1 foot-Lambert.

how can it be that higher values mean less contrast?

If the background is brighter, and the stimulus to be detected is a flash of light, then the brighter the background the less the flash will stand out: less contrast between the flash and the background.