How to detect the maximum resolvable spatial frequency of camera?

Cameras are typically poorly lowpass filtered (not adhering to Nyquist). Thus, small displacements in camera could mean large changes to the amplitude of a line that can barely be resolved.

The slanted edge method tries to estimate the limits of the optical system despite limitations in spatial sampling.

The image that you have obtained is already capturing the Modulation Transfer Function (MTF) of the optical system under test. It does not need to be further passed via an extra Discrete Fourier Transform (DFT) step. The spatial resolution limit is a matter of overall specification.

The test pattern's spatial resolution is measured in lines per millimeter and it varies in (a typical) $x$ direction (the width of the image) increasing towards the right.

As the resolution is increasing, the contrast (between the dark and light areas of the pattern) is decreasing.

Therefore, the simplest way to obtain the MTF is to create a scatter plot whose $x$ axis is spatial resolution in 1, 2,3,... lines/mm and the $y$ axis is the contrast (or modulation factor for that particular frequency) as $frac{I_{max}-I_{min}}{I_{max}+I_{min}}$.

BUT, as you can see, the difference between $I_{max}, I_{min}$ in the test image is dropping to zero (blue trace in your diagram). Therefore, the test pattern (image) itself does not represent a $1$ input to the optical system across all spatial frequencies.

Therefore, it would be best to calculate the MTF of the test pattern and then the MTF of the system under test and present both of them on the same diagram if you want to depict how close to the "ideal" the optical system under test really is. This is the point of the MTF.

To find the smallest object that this optical system can resolve, you need to translate the size of that object into lines/mm and check that the MTF at that limit is above the noise floor of the sensing element.

Hope this helps.