One of my mentoring students sent me a really great question today which I thought was such a useful lesson that I wanted to share it with everyone in my community…
This is important because getting this wrong risks deluding yourself about how good your trading system is (at best)… At worst it risks you loosing huge amounts of money before you realise something is wrong.
Here is the question…
“When I backtest my trading system on the list of US Stocks that are Optionable, I get a significant performance improvement… should I add that as a filter to my rules?”
Now before you say “yeah yeah, I have know all about survivorship bias”, please read on because the problem and solution is might be more subtle than you might think…
The problem is, however, when you backtest with certain lists of stocks, you are actually cheating by giving your system advanced knowledge that certain stocks will do well.
Why is this and which stock lists are a problem?
The problem is called Survivorship Bias, and the most obvious trap people fall into is backtesting their system on the current list of S&P500 stocks.
If you backtest your long side system over the last 30 years on the current list of S&P500 stocks, then you have a real problem because the stocks that make up the index today were not necessarily in the index 30 years ago, and the stocks that were in the index 30 years ago may not be in it today.
The impact of this is HUGE in backtesting because if you test your rules over the current list of S&P500 stocks then you are ONLY testing on stocks which SURVIVED and QUALIFIED to be in the index in the present day… but 30 years ago when your backtest ‘started’ you could not have possibly known which stocks would grow and become the 500 biggest stocks on the market and which ones would have failed…
Do you remember the movie ‘Back to the future”?
Maybe you are a geek like me and loved that movie… just in case you haven’t seen it *gasp* (or maybe I am just getting old… it was released in 1985) let me explain.
In the movie the main character Marty (Michael J Fox) uses a time machine called the DeLorean to travel back and forward in time. He ends up causing all sorts of unintended consequences, has emotional highs and has to deal with disasters of his own making.
Anyway, the key lessons are
- If you haven’t watched the movie, you should… and
- Time travel has unintended consequences which are not good
When you backtest using a list of stocks that is subject to survivorship bias, you are effectively giving your backtest the gift of time travel.
I usually test from 1990 or 1992 depending on which historical stock data I am using. So 30 years ago when your backtest starts Facebook was not part of the S&P500 yet… in fact Facebook didn’t even exist back then… but by running your backtest on the current day index constituents of the S&P500 you are giving your backtest advance knowledge that this stock is going to be a winner! In fact Facebook didn’t enter the S&P500 until 17 May 2012.
The distorting effect of giving your trading system this advanced knowledge may not cause the collapse of social order and alter history like Marty did in Back To The Future, but it will cause you to wrongly think you have an AMAZING trading system on your hands.
AND THAT COULD BE FINANCIALLY DEVASTATING!
After all, how could your system NOT make money if you knew in advance which stocks would grow to become the biggest stocks in the market 30 years from now!
With that type of time travel you would be guaranteed to make a fortune, and that is exactly what your flawed backtest will show if you backtest your system using the current list of S&P500 stocks.
What is Survivorship Bias in Trading System Development?
Survivorship bias a trading mistake that is caused by backtesting using a list of stocks that you would not have been able to choose at the time the trade was taken given the information that would have been available to you as a trader on the date the trade was taken.
OK – Now we all know what survivorship bias is…
When is it a problem and what can you do about it?
Let’s tackle the ‘when’ first
When is Survivorship Bias a risk in Trading System Development?
Survivorship bias is potentially a problem any time you backtest on a list of stocks that can change over time due to the size and liquidity of the companies on the list. For example:
- Any list of companies currently in any stock index: S&P500, Dow30, Nasdaq 100, ASX200 etc. These stocks were not always in the index, at one time they were just small caps with growth aspirations.
- Stocks that are currently Optionable: Investopedia says that “An optionable stock is one where the stock has the necessary liquidity such that a market maker, like a bank or an accredited financial institution, lists that stock’s options for trading.” Clearly a new company that has just been listed is unlikely to be optionable, so backtesting with the list of currently optionable stocks gives rise to survivorship bias and unachievable results.
- Stocks that are currently Shortable: For a stock to be shortable it must be borrow-able… for a stock to be borrow-able it must be held in fairly large quantities by mutual funds and large investors, which means it is probably a fairly popular stock. Just because a stock is shortable today does not mean it was big enough to be shortable in the past.
- Stocks that migrate along the value chain: If you backtest your system on a current list of Gold Producers then you WILL have a problem, because a company that is a large gold producer today might have been a microcap gold explorer 30 years ago… 10 years ago they may have found gold and gone into production and changed from an explorer to a producer. As a result the share price would have risen dramatically. So preselecting producers would cause you a survivorship bias problem too!
Backtesting with watchlists like the above will give you unrealistically good results because the system has advanced knowledge of what will happen with the stocks before it was possible to know that information.
But not all types of watchlists of cause this problem. It is usually only watchlists that stocks have to somehow ‘qualify’ to be part of, where that qualification could change in the future depending on the share price and liquidity of the company.
Other watchlists like Industry / Sector watchlists do not really have this problem. If you just backtest your trading system on Gold stocks for instance you won’t have this problem because a gold stock is a gold stock…
As a general rule industry and sector lists don’t cause a problem, but any list that changes over time due to company size, liquidity, profitability or share price will cause a problem.
What is the solution to Survivorship Bias In Backtesting?
The easiest solution is to not get into the problem in the first place… This means designing and backtesting your stock trading system using lists of stocks that are not prone to survivorship bias.
For this solution you can backtest over either:
- The entire market
- All stocks on a certain exchange (all Nasdaq stocks for example)
- Stocks in a certain industry classification (provided you are mindful of the gold example above)
How to backtest your trading system on stocks in a stock index
There is only one provider that I know of and recommend that has this data – Norgate Data.
Norgate’s data service is excellent:
- It is easy to install
- You can set it to perform updates automatically
- It manages stock splits and corporate actions automatically
- It provides historically accurate index constituents for ASX and US Stocks
(Sadly not other exchanges are currently covered)
- They provides full delisted stock history back to the early 1990’s
- Their data integrates seamlessly with Amibroker for backtesting and managing your trading systems