Why can't my Nikon D7500 control a Nikon SB-900 attached to the hot shoe via the 'Flash Control mode (external)' menu?

You're mistaken. The SB-900 is fully compatible with the D7500. Just because the flash doesn't do what you think it should do does not mean it's not compatible. Miscommunication between the camera and flash means that you could have the flash mis-seated in the hotshoe; the contacts/pins might be broken/dirty, the wires inside the flash might be broken; or the microswitch that tells the camera a flash is mounted on the hotshoe may be broken. IOW, it may be your copy of the speedlight or your camera hotshoe, not the model of flash that's the issue.

Pre-flashes can mostly be turned off if you don't use TTL. But TTL requires a metering pre-flash to work. And Nikon's wireless command mode requires pre-flashes to communicate settings changes. You can use an IR filter over the pop-up flash, or switch to radio triggering to turn that off, but any 3rd-party flash with CLS optical master capability would be exhibiting the same behavior.

Just me, but switching from a Nikon speedlight to a 3rd-party speedlight (even a Metz) and expecting better compatibility is not being particularly realistic. 3rd-party manufacturers typically reverse-engineer the electronic communication and as a result of that, are most often victim to future incompatibility (which is why so many 3rd-party flashes have firwmare upgrade features).

In addition to this, Metz Mecatech went out of business last year (2020). Just saying.

Thanks for the advice.

I will check again the latest software updates for the flash and camera. However, I will not be able to do it until the end the June, because I do not have this flash with me.

I hope your advice will be helpful.

Below you can find the full email I received from Nikon Europe support.

Case #: 04476320
Subject: Nikon D7500 with SB-900 (Flash control function)
Dear Grzegorz,

Thank you for contacting Nikon Product Support.

We are sorry to hear about your issue between the D7500 and the SB-900.

As we could confirm, the SB-900 cannot be controlled by the camera body using the "flash control mode". Unfortunately, and although we always advise to have the latest firmware, this will not solve the issue. 
Your suggestion/request of having a firmware update that allows you to control the flash on the camera body has been registered, with hope that Nikon may launch that update.

Besides the camera's user manual, you can check the options regarding speedlights, on the advanced speedlight compatibility chart (below), under unified flash control line, after selecting the D7500.

https://www.nikonimgsupport.com/eu/BV_article?articleNo=000006312&lang=en_GB&setRedirect=true

Hope this information helps to clarify your query.

Should you need any further assistance, do not hesitate to contact us.

Kind regards,

I will post an update on this issue after upgrading software (if newer will be available).