How to Invest: Forget the Needle, Buy the Haystack

Individual and professional investors alike have been underperforming benchmark indices for decades. Explanations brought forward include Efficient Market Hypothesis, behavioral biases, trading costs, and timing missteps. While those are possible sources of underperformance, another surprising phenomenon seems to be the culprit.
online investment

Serious business man trader analyst looking at computer monitor, investor broker analyzing indexes, financial chart trading online investment data on cryptocurrency stock market graph on pc screen. © Ground Picture /

October 21, 2023 05:37 EDT

A recent JPMorgan study has revealed surprising insights. It analyzed returns of various asset classes over a 20-year period. The study found that investors underperform the market. Importantly, this includes both individual and professional investors.

Before examining the reasons for this underperformance, it is important to look at the numbers. The average investor achieved annualized returns of 3.6% over 20 years. The Standard and Poor’s 500 (S&P 500 — an index comprising stocks of the 500 largest companies listed on US stock exchanges) achieved annualized returns of 9.5% over the same period. Even bonds, units of debt issued by governments and companies, gave returns of 4.3% over 20 years. 

Generally, investors put their money into both stocks and bonds. A 60/40-ratio of stocks and bonds would have returned 7.4% annually, while a 40/60-mix would have yielded 6.4%. Older investors are risk averse and often favor bonds over stocks because of guaranteed returns. In contrast, stocks can fall dramatically and, at times, lose all value.

With annualized returns of 3.6%, the average investor was able to double his or her money. The 60/40 stock-bond ratio should have led to 4.2 times increase in wealth while a 40/60 mix should have led to 3.4 multiple. Charles-Henry Monchau, the chief investment officer at Swiss Group Syz, estimates 95% of individual investors underperform the market. After fees and commissions, that number might be closer to 100%. Note this means that, except for a tiny percentage, investors almost invariably underperform the market.

In dollar figures, individual investors have left a lot of money on the table. From the end of 2018 to the same time in 2021 the S&P 500 rose by 90%. At year-end 2018, individual investors held equities worth $26.7 trillion. The annualized return figures tell us that individual investors missed out on gains of $3.6 to $5.9 trillion. What is going on?

Underperformance leads to the rise of passive investing

It makes sense that individual investors underperform the market. They do not have the same information as professional and institutional investors. They suffer from information asymmetry. These investors also fear losses, chase market darlings (stocks often discussed at dinner parties and rarely questioned as a good investment), chase hyped-up companies, fail to time the market and make other mistakes that individuals often do when investing or trading alone.

Surprisingly, professional and institutional investors do not outperform the market either. A study by S&P Dow Jones Indices reveals that up to 96% of all active US equity funds underperformed their benchmarks over a 15-year period. Note that only 30-60% of these funds survived over this period. Most underperforming funds simply closed shop. Some merged with others. So, there is a survivorship bias — a type of selection bias that ignores the unsuccessful outcomes of a selection process — to this 96% figure. The real figure is even higher.

So, why are individual and professional investors struggling to beat the market? After all, an index is a mix of companies of varying quality — some are great, some mediocre and some outright bad.

There is a logical problem with the idea of investors beating the market. Very simply, the market is nothing but all the investors buying and selling to each other. For any trade in the market, one investor has to sell to another. For every investor outperforming the market, another has to underperform.

Over the years, passive investors have emerged. These are exchange-traded funds (ETFs) that contain hundreds — sometimes thousands — of stocks or bonds listed on the market. Their basket of stocks or bonds closely follow the performance of the index neither out- nor underperforming the market.

In recent years, passive investing is rising. As a result, the number of market participants who still can under- or outperform is shrinking. According to Bloomberg data, more than 54% of all assets in US equity mutual funds and ETFs are now managed passively.

Who outperforms the market and why?

As most individual and institutional investors are underperforming, who then is outperforming the market? 

Outperformers tend to be hedge funds, activist and quantitative investors, insurance companies, pension funds and conglomerates like Berkshire Hathaway. 

Hedge funds use strategies usually not available to individual investors or mutual funds, such as leverage, arbitrage, combination of long and short positions, derivatives, and algorithmic trading. Activist investors take concentrated positions in companies to force management or strategic changes, which is impossible for individual investors due to lack of size. Unlike hedge funds, mutual funds usually do not take a combative stance towards company boards.

Quantitative investors use mathematical models and computer algorithms to exploit patterns and trends in financial markets. Individual and most institutional investors do not have access to trading technology to enter and resell positions within fractions of a second. Insurance companies and pension funds can afford to ignore short-term market turmoil as their capital is usually of long-term nature. Conglomerates like Berkshire Hathaway get a detailed look into the accounts of a potential takeover target before an acquisition, receiving better information than what individual shareholders obtain via quarterly and annual reports.

Note that this long-term advantage of pension funds might be lost as many outsource management of their assets. According to a BNY Mellon study, 50% of the largest public asset management companies exclusively use external managers. These managers tend to take short-term, not long-term decisions. Furthermore, these institutions often suffer from poor governance. This can have a detrimental impact on their performance.  Compared to their Canadian peers, “American public pension funds are stuffed with politicians, cronies and union hacks” and tend to perform more poorly.

A simpler reason for why it is so hard to beat, or even match, the performance of benchmark indices like the S&P 500 lies in a skewed distribution of returns. The performance of index member companies is not normally distributed (which would follow the bell curve) but has a huge right “fat tail”. Simply put, only a few companies have astonishingly outsized returns. Not owning those few companies automatically leads to underperforming the index.

Between 1995 and 2022, only ten stocks (just 2% of 500 companies) accounted for at least one-fifth of the performance of the S&P 500. In some years, the top ten stocks provided more than 100% of index performance. This means if we exclude these ten stocks, the S&P 500 would have had a negative return. 

Over the first nine months of 2023, the “magnificent seven” — Apple, Amazon, Alphabet, Meta, Microsoft, Nvidia, and Tesla — are up 92%, while the remaining 493 members of the S&P 500 have gained only 3%. Seven out of 500 are tiny odds. And it would not have been sufficient to simply own those stocks — one would have had to own them in the same ratio as the index. Those “magnificent seven” stocks currently command a 28% share of the index. Those who want to match the S&P 500 would have to have the same weighted portfolio. This means, investors would have to heavily invest in technology stocks precisely when the sector commands historically high valuations. 

Only a few winners emerge from the large number of companies listed on the market. Out of 28,114 publicly-listed US companies, the top 25 (less than 0.1%) are responsible for nearly one-third of all shareholder wealth created since 1926. These numbers underline the fact that the odds of picking those few massive outperforming stocks are very slim. Almost invariably, stock picking turns out to be a losing proposition.

Stock indices are simply less risky

Apple went public in December 1980 at a price of $22. Adjusted for stock splits, its initial public offering (IPO) price was $0.10 per share. At today’s price of $179.49, Apple has since gained 179,390%, a 19% annualized return. Yet to get that return, an investor would have had to sit through multiple difficult periods. From 1991 to 1998, Apple’s stock price declined by 83%, from 2000 to 2003 by 82% and from 2007 to 2009 by 61%.

Any potential Apple investor would have had to shrug off negative news headlines, like this one — “The Fall of Steve Jobs” — from Fortune Magazine in 1985. Shortly after this story, Jobs was fired from Apple. He returned 12 years later and led the company to great success. However, it would have taken a brave and stubborn investor to hold on to Apple stock and they would have had to refrain from taking any profits for a long period of time.

Asked which stock investors wished they had bought (in hindsight), most would likely name Apple or Tesla. However, a beverage company, emerging out of bankruptcy in 1988, steals the crown. Formerly known as Hansen Natural, Monster Beverage rose from a split-adjusted price of $0.0062 in 1995 to around $50 today, for a return of more than 800,000%. The annualized gain of 37% for Monster Beverage is almost twice Apple’s 19%.

There is no guarantee that companies can come back from steep declines in stock prices. In such cases, investors’ stubbornness can backfire. The share prices of former market leaders were nearly or completely wiped out. Former stock market darlings such as Nokia (-90%), Palm (-94%), Blackberry (-98%) and Nortel Networks (-100%) are part of a long list of companies that have sunk like lead in water.

In the case of an investment manager, he would have been fired for holding on to Apple or Nokia stock. Holding an ETF saves professional fund managers from the risk of losing their jobs.

Jack Bogle, the founder of investment management company Vanguard, famously exclaimed “Don’t look for the needle in the haystack. Just buy the entire haystack.” The “needles” investors are looking for are the few companies whose shares go on to have an astronomically high performance. The “haystack” is the entire stock index. Vanguard introduced a low-cost index fund in 1976, leading to the success of the ETF. Not only does an index ETF guarantee to closely follow the market but it does so at very low cost. The Vanguard S&P 500 ETF charges 0.03 and even the State Street Global Advisors “SPY” ETF charges 0.09%, a much lower figure than active professional and institutional investors.

What if everyone goes passive?

From a rational perspective it does not make sense to spend millions of dollars on salaries of analysts and portfolio managers if the prospects of outperforming a simple (and cost-efficient) ETF are slim. So what would happen if most investors shifted to passive investing via index-linked vehicles? What if nobody did any research anymore into companies’ fundamentals, balance sheets and products? 

Index members could rely on steady buy orders from automatic investing by pension funds and insurance companies. However, this raises other issues. Would the stock price of a company reflect the fact that it was on the verge of bankruptcy or that it had just invented a cure for cancer? Would the price mechanism of the market still work?

In theory, there must be a maximum share of passively managed money beyond which active investing would become profitable again. But the fundamental conundrum of the market would still remain: for every investor that outperforms there must be another who underperforms the index.

For individual investors, going passive does by no means guarantee investment success. Passive investing simply means no underperformance relative to an index but does not guarantee absolute (positive) performance. It took the technology-heavy Nasdaq Composite 15 years to recuperate losses after the dot-com bubble burst in 2000. Between 1995 and its peak in March 2000, this index rose 800%, only to give back most of its gains by October 2002.

Today, S&P 500 heavyweights such as Apple and Microsoft are valued together at over $5 trillion. They are sporting historically high valuations with their valuations at 28 and 30 times their estimated earnings respectively. The index containing these stocks doesn’t care about the valuations of Apple and Microsoft. The S&P 500 does not care about the future performance of Apple and Microsoft. Passive investing can solve relative underperformance vis-à-vis the market but not guarantee high returns because, like the Nasdaq in 2000 or Wall Street in 1929, the market itself can lose value.

The views expressed in this article are the author’s own and do not necessarily reflect Fair Observer’s editorial policy.


Only Fair Observer members can comment. Please login to comment.

Leave a comment

Support Fair Observer

We rely on your support for our independence, diversity and quality.

For more than 10 years, Fair Observer has been free, fair and independent. No billionaire owns us, no advertisers control us. We are a reader-supported nonprofit. Unlike many other publications, we keep our content free for readers regardless of where they live or whether they can afford to pay. We have no paywalls and no ads.

In the post-truth era of fake news, echo chambers and filter bubbles, we publish a plurality of perspectives from around the world. Anyone can publish with us, but everyone goes through a rigorous editorial process. So, you get fact-checked, well-reasoned content instead of noise.

We publish 2,500+ voices from 90+ countries. We also conduct education and training programs on subjects ranging from digital media and journalism to writing and critical thinking. This doesn’t come cheap. Servers, editors, trainers and web developers cost money.
Please consider supporting us on a regular basis as a recurring donor or a sustaining member.

Will you support FO’s journalism?

We rely on your support for our independence, diversity and quality.

Donation Cycle

Donation Amount

The IRS recognizes Fair Observer as a section 501(c)(3) registered public charity (EIN: 46-4070943), enabling you to claim a tax deduction.

Make Sense of the World

Unique Insights from 2,500+ Contributors in 90+ Countries

Support Fair Observer

Support Fair Observer by becoming a sustaining member

Become a Member