Do you need to remove buildings to remove cultural penalty?

In Total War: Remastered, cities quite close to my capital are slowly getting angry because of cultural penalties and squalor.

In this post, I am mostly curious how to get rid of the cultural penalty. Do I really need delete the buildings that have a cultural penalty and then rebuild the Roman alternative? In some cities this effect is -45% so it is quite a big deal.