Town vs City, why they can be used interchangeably in novel?

I know that this is a simple question, but it does made me wonder. This is what I found after reading quite a lot of novels and often happened:
Why does City of something was sometimes called town when other people speak? Why they not just stay on saying a city, a "city" and instead often change it to "town" as they see fit.
Does town here specified something in particular and that’s why they were used instead of city like how it should be?