Statistical difference in DEM-height compared to a specific point(feature)

Recently I’ve been struggling when trying to compare surrounding points of a DEM to a specific one (a feature in my case. A castle to be precise). I want to find out how steep a slope in an area around the castle is, compared to the castle/point itself.

I’ve tried with a buffer for the area and using zonal statistics afterwards, but this has proven to be unsatisfying for my research question.

Anyone got a smart approach for this?