Is the S&P 500 really the best index/vehicle for long-term investing?

Nearly every source I’ve read or heard has said that for long-term investing, the best option is to invest in an ETF based on the S&P 500 Index. The logic is that, over the course of many decades, there are almost no investment strategies that have proven to yield a higher average total return (subtracting fees and taxes) than the S&P 500.

As far as I’m aware, the S&P 500 index is mathematically quite simple – straight-forward weighting of the top 500 (or so) stocks based on free-float market capitalization. I find it extremely surprising that out of all the complexity of the market, the best performing strategy (in the long run) would be the S&P 500.

For example, by looking at the numbers, it seems that the S&P 600 index, which follows the same weighting formula using the same class of stocks but only using "small-cap" American stocks, has produced a better long-term total return than the S&P 500. The average total annual return over the years 2004-2020 for the S&P 500 was 10.9%, whereas for the S&P 600 it was 12.9%. I understand the S&P 600 is more volatile due to its small-cap focus, but in the long-term it clearly seems to be the better option, right?

It seems to me that a slight shift of focus to small-cap stocks (e.g. S&P 600) generates better long-term returns than focusing only on large-cap stocks (e.g. S&P 500). To me this seems like a very obvious (and naive?) guess – smaller companies have more room to grow. But if this is true, how does it not directly contradict with the common saying that nothing beats the S&P 500 in the long term?