One of my mentoring students sent me a really great question today which I thought was such a useful lesson that I wanted to share it with everyone in my community… 

This is important because getting this wrong risks deluding yourself about how good your trading system is (at best)… At worst it risks you loosing huge amounts of money before you realise something is wrong.Β 

Here is the question…

β€œWhen I backtest my trading system on the list of US Stocks that are Optionable, I get a significant performance improvement… should I add that as a filter to my rules?”

Now before you say β€œyeah yeah, I have know all about survivorship bias”, please read on because the problem and solution is might be more subtle than you might think…

My student is doing super well and has developed and refined a trading system which they are already happy with using the Methodology I teach inΒ The Trader Success System. At the moment I am helping him put the finishing touches on it and test any final improvement ideas that come up during brainstorming… so the question is a very natural one at this point in the process.

The problem is, however, when you backtest with certain lists of stocks, you are actually cheating by giving your system advanced knowledge that certain stocks will do well.

Why is this and which stock lists are a problem?

The problem is called Survivorship Bias, and the most obvious trap people fall into is backtesting their system on the current list of S&P500 stocks.Β 

If you backtest your long side system over the last 30 years on the current list of S&P500 stocks, then you have a real problem because the stocks that make up the index today were not necessarily in the index 30 years ago, and the stocks that were in the index 30 years ago may not be in it today.

The impact of this is HUGE in backtesting because if you test your rules over the current list of S&P500 stocks then you are ONLY testing on stocks which SURVIVED and QUALIFIED to be in the index in the present day… but 30 years ago when your backtest β€˜started’ you could not have possibly known which stocks would grow and become the 500 biggest stocks on the market and which ones would have failed… 

Do you remember the movie β€˜Back to the future”?
Stock trading systems – are you fooling yourself with faulty backtests? - back to the future

Maybe you are a geek like me and loved that movie… just in case you haven’t seen it *gasp* (or maybe I am just getting old… it was released in 1985) let me explain.

In the movie the main character Marty (Michael J Fox) uses a time machine called the DeLorean to travel back and forward in time. He ends up causing all sorts of unintended consequences, has emotional highs and has to deal with disasters of his own making.

Anyway, the key lessons are

  1. If you haven’t watched the movie, you should… andΒ 
  2. Time travel has unintended consequences which are not good

When you backtest using a list of stocks that is subject to survivorship bias, you are effectively giving your backtest the gift of time travel.Β 

I usually test from 1990 or 1992 depending on which historical stock data I am using. So 30 years ago when your backtest starts Facebook was not part of the S&P500 yet… in fact Facebook didn’t even exist back then… but by running your backtest on the current day index constituents of the S&P500 you are giving your backtest advance knowledge that this stock is going to be a winner!Β In fact Facebook didn’t enter the S&P500 until 17 May 2012.

The distorting effect of giving your trading system this advanced knowledge may not cause the collapse of social order and alter history like Marty did in Back To The Future, but it will cause you to wrongly think you have an AMAZING trading system on your hands.

AND THAT COULD BE FINANCIALLY DEVASTATING!

After all, how could your system NOT make money if you knew in advance which stocks would grow to become the biggest stocks in the market 30 years from now!

With that type of time travel you would be guaranteed to make a fortune, and that is exactly what your flawed backtest will show if you backtest your system using the current list of S&P500 stocks.

What is Survivorship Bias in Trading System Development?

Survivorship bias a trading mistake that is caused by backtesting using a list of stocks that you would not have been able to choose at the time the trade was taken given the information that would have been available to you as a trader on the date the trade was taken.

OK – Now we all know what survivorship bias is…

When is it a problem and what can you do about it?

Let’s tackle the ‘when’Β first

When is Survivorship Bias a risk in Trading System Development?

Survivorship bias is potentially a problem any time you backtest on a list of stocks that can change over time due to the size and liquidity of the companies on the list. For example:

  1. Any list of companiesΒ currentlyΒ in any stock index:Β S&P500,Β Dow30, Nasdaq 100, ASX200 etc. These stocks were not always in the index, at one time they were just small caps with growth aspirations.
  2. Stocks that areΒ currentlyΒ Optionable:Β Investopedia says that “An optionable stock is one where the stock has the necessary liquidity such that a market maker, like a bank or an accredited financial institution, lists that stock’s options for trading.” Clearly a new company that has just been listed is unlikely to be optionable, so backtesting with the list ofΒ currentlyΒ optionable stocks gives rise to survivorship bias and unachievable results.
  3. Stocks that areΒ currentlyΒ Shortable:Β For a stock to be shortable it must be borrow-able… for a stock to be borrow-able it must be held in fairly large quantities by mutual funds and large investors, which means it is probably a fairly popular stock. Just because a stock is shortable today does not mean it was big enough to be shortable in the past.
  4. Stocks that migrate along the value chain:Β If you backtest your system on a current list of Gold Producers then you WILL have a problem, because a company that is a large gold producer today might have been a microcap gold explorer 30 years ago… 10 years ago they may have found gold and gone into production and changed from an explorer to a producer. As a result the share price would have risen dramatically. So preselecting producers would cause you a survivorship bias problem too!

Backtesting with watchlists like the above will give you unrealistically good results because the system has advanced knowledge of what will happen with the stocks before it was possible to know that information.

But not all types of watchlists of cause this problem. It is usually only watchlists that stocks have to somehow ‘qualify’ to be part of, where that qualification could change in the future depending on the share price and liquidity of the company.

Other watchlists like Industry / Sector watchlists do not really have this problem. If you just backtest your trading system on Gold stocks for instance you won’t have this problem because a gold stock is a gold stock…

As a general rule industry and sector lists don’t cause a problem, but any list that changes over time due to company size, liquidity, profitability or share price will cause a problem.

What is the solution to Survivorship Bias In Backtesting?

The easiest solution is to not get into the problem in the first place… This means designing and backtesting your stock trading system using lists of stocks that are not prone to survivorship bias.Β 

For this solution you can backtest over either:

  1. The entire marketΒ 
  2. All stocks on a certain exchange (all Nasdaq stocks for example)
  3. Stocks in a certain industry classification (provided you are mindful of the gold example above)

How to backtest your trading system on stocks in a stock index

If you really want to backtest your stocks on watchlists that do have changes in their constituents over time (Like the stock indices) then you need to purchase data which has historically accurate index constituents.Β 

There is only one provider that I know of and recommend that has this data –Β Norgate Data.Β 

Norgate’s data service is excellent:

  • It is easy to install
  • You can set it to perform updates automatically
  • It manages stock splits and corporate actions automaticallyΒ 
  • It provides historically accurate index constituents for ASX and US Stocks
    (Sadly not other exchanges are currently covered)
  • They provides full delisted stock history back to the early 1990’s
  • Their data integrates seamlessly with Amibroker for backtesting and managing your trading systems

With just a few additional lines of code in your Amibroker trading system, usingΒ Norgate dataΒ you can backtest your trading system on historically accurate index constituents AND delisted stocks.Β 

If you want to backtest your stock trading system on historically accurate index constituents, check out Norgate’s offeringΒ – you won’t be disappointed.

Pin It on Pinterest

Share This

Share This

Share this post with your friends!