Why speaking about diffeomorphisms for change of coordinates?

This is a question I've also often gotten confused over. (For this answer I don't see the need to speak about charts and maps.) The way I see it is that often when Physicists talk about diffeomorphisms they really just mean a coordinate transformation. However as far as I'm aware, when considering diffeomorphisms you're looking at how tensors change under a pushforward and a coordinate transformation. E.g. for a tensor, we apply a pushforward to the coordinates of the tensor, evaluate the tensor at the point, and then make a transformation back to the original coordinates. I'm unsure if this is equivalent to what people mean by an active coordinate transformation (i.e. the physical spatial points change but the coordinates do not).

The key distinction between a coordinate transformation, in my opinion, is that for a diffeo the coordinates don't change, therefore the volume element $d^4x$ doesn't change, but $sqrt{-g} d^4x$ would. Conversely, we know for a coordinate transformation it's the other way round.

This is the source I used, but I'm not sure how consistent this is with other text. E.g. Carroll says that diffeo's are an 'active' view and standard coord transformations are a passive view, but they're practically the same: we move the points on a manifold under the diffeo and then evaluate the coords of the new points. Wald instead talks about both active and passive diffeos, and says they're philosophically different but totally equivalent.