Creating an "unclassified" class in Random Forest

I am trying to classify satellite based images by creating a region of interest and then classifying according to it.

I am using a Jupyter notebook using python to do that.

I used a Random forest classifier and got a nice model and result, but the problem is that the image is "over classified" meaning that all the pixels et value and force to be classified.

I would like to define level of similarity that a pixel has to have in order to be classified, otherwise, it will not get any class.

For example, the black suppose to be asphalt:

However, in the RGB, you can see it’s not asphalt:

Is there any way to define in random forest or any other algorithm "level os similarity"? (For example something similar to n-D angle to match pixels to reference like ised in SAM, but under random forest, or another algorithm that allows define that)

SAM- https://www.harrisgeospatial.com/docs/SpectralAngleMapper.html

My end goal: to get "unclassified" values based on similarity level to the calibration data