What does “summer and winter” mean?

I’d like to ask about the sentence from Lion’s Mane by Conan Doyle.

Summer and winter he went for his swim, and, as I am a swimmer myself,
I have often joined him.

This “he” is Mr. McPherson, science lecturer who lives in this coastal area, and “I” is Sherlock Holmes.

I don’t know how to interpret the phrase in the beginning of the above sentence, “summer and winter”. This literally means Mr. McPherson comes to the beach to swim in summer and winter, and not in spring and autumn right? But could this phrase mean all seasons? If it’s “summer or winter”, I know it says that he comes to swim in summer and winter, not to mention spring and autumn. But I don’t know.
Isn’t it a bit strange for someone to choose winter for swimming among four seasons? I thought like that his job as a teacher there may be busier in spring and autumn than the rest of the seasons, and summer and winter is best time for him, but he seems to go swimming before his classes start so his job doesn’t seem to affect in which season he chooses to swim.
Doesn’t this phrase function like this.. They regard summer as the best one winter the worst one to do some activity. Like in “rain or shine” rain maybe the worst case, and shine maybe the best case. So Mr. McPherson goes there not only in the best season (summer) but also in the worst season (winter), so you can be sure he goes in seasons coming between, hence he goes there in all year round.
The problem is, I couldn’t find “summer and winter” as an idiom, set phrase like “rain or shine” on the net.
Could someone tell me?