Imagine yourself back in July 2019. If you want to beat the S&P 500 index, obviously you need to predict the worst performing stocks, so you don't buy them.

Nobody knew anything about Covid-19 back then, remember.

But nine out of the ten worst performers (which all fell between 60% and 72% in a year) were all obvious victims of Covid, with hindsight: three cruise line companies, three oil and gas companies, two retailers, and one airline.

The odd one out (Xerox) just snuck in at 10th-worst (and down 58%) because it pulled out of its takeover bid to buy HP. And that was obviously easy to predict a year in advance - not.

Are you still wondering why beating the index in the long term is hard?

There are a lot of great answers here, but there's one additional dimension to this that I don't think has been touched upon heavily enough: Risk

---

When I was younger (early teens through mid 20s) I really enjoyed playing the stock market. I had a steady income, few financial responsibilities, and half a century to recover from any real losses. In that time period I bought a lot of different stocks, and outperformed the S&P by 3x-5x consistently. In not one single year, over the course of 10-12 years, did I do worse than the S&P, and I never lost money in any given year.

My first stock purchase was CMHHF for $0.11, and I sold it for $1.25 6 months later. I bought my first shares of TSLA for $18, GOOG at under $200, and so on.

Now that I'm 40 and have a family and retirement to worry about, I don't invest that way any more. I'm very glad that I did at the time, and I would again if I could go back in time, but there's a downside.

I had 3 major losses in that period. And I mean major. In fact, I had one company, that I had invested a few thousand in (that was a lot of money for me back then) go under. All (over 99.9%) of my money invested in that company...just gone. It was like the bank with my safety deposit box full of cash burned down and a few nickels survived the flame. This was during the same yearish period that TSLA jumped from 60ish to 180ish, so I was still up and didn't worry much.

But consider an alternate set of circumstances. What if instead of TSLA I had bought one or two other companies that also crashed or went bankrupt during that time? Especially since I had most of my money in just 3 or 4 stocks. I could have lost everything. A 1500% return over 3 years isn't too meaningful if you suffer a 99% loss the following year. $1000 becoming $15,000 is great, but if it then becomes $150...the only thing that matters is that you're 85% down over 5 years.

---

I think this plays a major role in your question, because when you're not thinking about risk you can employ strategies that can provide massive returns, and even do so consistently enough that it seems "a sure thing" (because our personal experience is a statistically insignificant sample size). However, it is not a sure thing. And as you age you have less free money, more financial responsibilities, and far more pressure to maintain a certain level of savings for retirement. That level of risk is no longer acceptable.

To put it another way, beating the S&P 500 is actually pretty easy, as you may have seen from experience. Beating it with enough consistency to be a viable strategy for long-term goals; that's difficult! And since most investors fall into that latter category, I think they often forget that there are investors out there with youth and plenty of disposable income on their side whose risk tolerance is through the roof, and thus claim that it's hard to do without the caveat of "with acceptable risk".

How to outperform the market

TL;DR: Know something others don't.

• Identify a handful of small-cap stocks. Some of them will be undervalued, some overvalued. But which?
• Get to understand their business. In detail.
• Go through their books.
• Talk to their directors. Talk to their staff.
• Understand their market. Talk to their customers.

Most small-cap stocks don't receive this amount of attention. So you now are likely to have a better idea which businesses are likely to do well over the coming years than the market.

But:

• It is time-consuming, skilled work.
• Information costs. To know more than the market you had to spend more time and attention than the market did.
• Understanding costs. You had to research the market, the business, and then you had to put it all together, which requires time, skill and experience. Experience costs.

Accounting for your time, and the opportunity cost of the other work you could have been doing, you may not come out ahead of the market.

That's why you may well be the only person doing it for some of these businesses. It's a rational decision by fund managers to balance the cost of the analysis against their returns.

This is a little bit like saying "why do people bet on the favorites in sporting events" and then citing the results of some sporting events where a bigger return could have been made if you knew the result in advance.

I mean the short answer is, people don't have time machines. Making money on the stock market is an exercise in predicting the future so when people say stick with the S&P 500 what they really mean is it's a safer investment than going for higher risk/high reward gambles on other companies.

Buy Netflix, you say? Solid business, guaranteed to beat the market?

Perhaps you didn't read today's news?

• Netflix shocks Wall Street with earnings miss, weak 3rd quarter guidance
• Netflix Q2 earnings miss expectations as competition increases from Apple, Amazon, and TikTok
• Shares Of Netflix Smacked After Earnings Miss & Lowering Guidance

Past results are no guarantee of future performance. This is why it's hard to beat the market: the future is difficult to predict. Is this just a little blip in continued Netflix growth? Or is it the beginning of the end of Netflix? It's hard to say, but regardless, if you'd bought Netflix when this question was posted, you would have lost about 10% overnight.

That's not to say no one should try: if a business looks good, it should be worth more. Turns out Netflix wasn't as good as people thought it was, and that's why the price went down.

If you want to put most of your money into business that most people agree are doing well and are valuable, you can save yourself some work and buy a capitalization weighted index fund.

Whereas companies like Google, Facebook, Netflix, Amazon, Apple have all increased anywhere from 3x-6x in value in the past 5 years.

In fact, if you replace Netflix with Microsoft in this list, you've named the five largest components of the S&P 500 index. If you put some money in an S&P 500 index, 20% of your investment would go into Google, Facebook, Amazon, Apple, and Microsoft. If you invest in the top 40 companies, you've replicated half of the S&P 500.

So, "buy an index fund" and "just google the top 10 companies in the S&P 500 and invest in them" are not so different strategies.

The stocks you mentioned are pretty good. So is a Nasdaq index fund, which has had double the returns of the S&P 500 over the past 5 years.

The Nasdaq has a solid record of beating the S&P 500. If you'd invested in a Nasdaq index fund in 1997, by 2000 you'd have triple the gains of the S&P 500.

(^IXIC: Nadsaq; ^GSPC: S&P 500)

So it's February 2000. You buy a Nasdaq index fund, because Nasdaq is full of stocks that have been doing great.

Whoops. I guess all those Nasdaq stocks were doing so well because they were overvalued. But we know that now, and so we won't do that again, right? Right? Am I right?

One aspect of the misunderstanding that no one has really mentioned yet:

It seems like any reasonable person 5 years ago should have been able to realize that Google, Apple, et. al. would do well and buy accordingly.

But that's not at all true, and there are at least 2 reasons:

1. In order for a stock to do really well, it's NOT enough that the company does well.

Efficient market hypothesis: all publicly available information is already incorporated in to the price. So for the stock to go up it's not enough that the company meet its earnings projections. It has to do unexpectedly well. Now certainly, stock represents in a sense the real value of the company and it will go up some just because the company is growing, but that isn't going to let you beat the market (again, unless it is growing at an unexpected rate). To put it slightly differently: you can't beat the market just by doing something that would be obvious to any reasonable person, because those reasonable people have already driven the price up accordingly. A failure to understand this (admittedly somewhat counter-intuitive) point is probably one of the most common investing mistakes. I vividly remember my mother (a former stock broker) trying to explain this to a friend of hers (an otherwise sophisticated and savvy guy) when he was complaining that a stock he owned went down despite the company having it's most profitable quarter ever. Watching this obviously very intelligent guy struggle to get this point is what really drove home for me how alien it can be for some people.

So just because in 2015 you correctly predicted that Google and Apple (which were already big dominant companies at the time) would continue to do well for at least 5 more years, you couldn't necessarily predict that their stocks would continue to go up, much less shoot up.

2. The rosy scenario played out

This turf has been better covered by the other answers but, just like you could have been randomly hit by a bus yesterday and died instead of posting on the internet, something awful could have happened to one of those companies that didn't. Who could have predicted that mobile phone market leader Samsung would decide to ship an exploding phablet (to name a tech example sticking with the flavor of yours)? Looking back over what did happen gives the illusion of inevitability.

It really depends on what you mean by "hard to outperform the S&P 500".

Consider a roulette wheel. In a certain sense, it's easy to make money at a roulette wheel. The strategy is very simple: if the wheel is about to come up red, then bet on red, and if it's about to come up black, then bet on black. You'll double your money!

However, this strategy is obviously impossible to implement, because you don't know whether the wheel is going to come up red or black. Indeed, there is no winning strategy for roulette. Sure, you might win, but no matter what strategy you choose (aside from not betting at all), on average you'll lose.

The stock market is similar. It's very easy to come up with a strategy which might outperform the S&P 500. It's extremely difficult to come up with a strategy that will outperform the S&P 500 on average. Trading firms spend millions of dollars just to get a tiny edge.

Hindsight is 20:20.

companies like Google, Facebook, Netflix, Amazon, Apple have all increased anywhere from 3x-6x in value in the past 5 years.

...while at the same time, companies like General Electric, Walgreens or Kraft lost a considerable amount of their stock value.

How could you have known 5 years ago which companies would be "good" and which companies would be "bad"?

You couldn't.

Which is why buying an index fund is a great way to hedge your investments. You might not win as big, but you also won't lose as big by betting on the wrong horses.

Welcome new user, I will take a crack at explaining this. It is a good question and it's a very common misapprehension.

I think the shortest possible explanation is this:

You see how you said "... a top company such as Apple ..." ...

Forget 5 year. Here's a 25+ year chart of Apple:

It barely tripled in 20+ years. What an absolute dog!

Every day for some 25 years, people said "Apple is a 'good' 'top' 'best' company, simply invest in it to get rich." Many of those people literally died waiting for it to explode.

It's ironic that this question asserts "You need only invest in Apple to get rich!"

Apple is the prime example in 600 years of investing history where that idea flopped, with people waiting their whole lifetimes for it to pay off.

It can be difficult to grasp this (it's difficult for me, that's for sure!) but when we write a sentence such as "... a top company such as Apple ..." ...

... you are just saying: "a company which went up a lot the last few previous years".

It is "begging the question" in the biggest possible way.

From the question,

"You can even just google the top 10 companies in the S&P 500 and invest in them. The average gains from all of those companies .. [was big]

"The top 10 companies in the S&P 500" literally just means "the companies that gained the most the last few years!"

The two parts of the sentences are just: restating the same thing.

If you had used "your system" 5 or 10 years ago, it would not have been that same list.

TripleHound's pithy comment perfectly encapsulates the situation: just look at Pan-Am, Enron, Lehman Brothers.

(Recall that Apple stock was such a dog for so long, that at one point Steve Jobs, for goodness sake, dumped all his stock!)

Your basic trading idea ("invest in the currently biggest companies and hope they will grow") is as good as any other trading idea (ie: terrible! :) ) and it's been around a long time.

Back in the '70s folks would talk about the "Nifty Fifty" which was exactly the idea you're considering. Half the companies don't even exist any more. It's the classic standard tale of why "just invest in the currently biggest" doesn't work:

https://www.forbes.com/sites/investor/2019/05/24/are-any-of-the-nifty-50-stocks-still-nifty/#73b859441b17

(I.M.H.O. indeed the central mental goal of trading, on whatever time scale you do it, is to get away from the assumption that recent past performance indicates the future, you really have to completely let go of that mental state to trade.)

Thus,

"I am just curious at what I am missing..."

The answer is straightforward, you are making a severe logical error.

In the first three paragraphs you say "Companies A, B C have increased greatly in price the last X years. If you had invested in them X years ago, you would have made a lot of money."

Note that in making the list of A, B, C you refer to "good" companies, "major" companies and "top" companies.

But.

Good, major, top companies are literally just "those that have increased greatly in price" in the previous few years. Note the increased past tense.

If you could guess which companies are about to increase greatly in price in the next few years, you'd be the richest person in the galaxy.

Observe that in a few years from now, the companies that at that future time will be mentioned as good, major, top companies will be the ones that did, over our future, increase greatly in price.