Introduction to Backtesting

What is Backtesting in Trading

Backtesting is an essential technique in the trading world and is useful no matter which financial markets you trade. It involves applying your trading rules, or your trading strategy, to past data using backtesting software. This is essentially a historical simulation to show you how well your trading strategy has performed in the past. The objective is to simulate all the signals that your trading strategy would’ve generated in the past and assess their profitability. This allows you to determine the effectiveness of your trading strategy in real-time future trading.

Purpose of Backtesting

Backtesting’s primary role is to help you maximize the probability of positive returns from your trading strategy. It allows you to eliminate unsuccessful strategies by identifying that they lost money in the past. Moreover, backtesting aids in the elimination of fragile strategies by revealing their performance over a wide range of market conditions and parameter values.

The significance of backtesting is evident when faced with the question: “Does backtesting matter if it’s based on past data and future trading conditions could differ?” Consider this – would you prefer to trade a strategy that has shown consistent positive returns over 30 years of backtesting or one that has shown losses over the same period? Clearly, a strategy that lost money in the past is not likely to turn profitable just because you’ve decided to trade it. Backtesting, therefore, provides valuable insights to guide your future trading decisions across all financial markets and timeframes.

Benefits of Backtesting

Backtesting has several benefits that give you an edge in trading. First and foremost, it instils confidence. Knowing your strategy was profitable in the past could indicate potential future positive returns. It also deters you from adopting failing strategies, saving you from potential losses.

Furthermore, backtest simulations aid in understanding your strategy’s dynamics and historical performance. For example, gaining insights about the possible duration of your winners and losers, the average size of winnings and losses, and the frequency of wins and losses prepares you psychologically for future trades. This knowledge and understanding make it much easier to stick to your strategy as you are aware of what to expect. Essentially, backtesting prepares you for the realities of trading, thereby making it easier for you to follow your strategy and maintain confidence.

Limitations of Backtesting

Despite its benefits, backtesting has its limitations – Backtests are not predictive models that show you how rich you will become… The most significant limitation is that they rely on past data. Successful backtests do not guarantee future positive returns. It merely tilts the odds in your favor. The effectiveness of backtesting hinges heavily on its execution – there’s a vast difference between good and bad backtesting.

A common pitfall in backtesting is excessive optimization, where you might test numerous parameter combinations and choose the one that performed the historical performance. For example, if you tested 300,000 combinations of parameter values for a trading strategy and decided to trade the one that worked best over the past 20 years, you’re likely to be disappointed. The strategy might perform poorly in real-time trading because you didn’t validate the system’s stability over time, its robustness across various parameter values, or understand its volatility.

Therefore, good backtesting involves deep interrogation and investigation of your trading strategy. As a trader, the more you understand about how your strategy will work, the more confident you’ll be, and the more likely you are to spot potential problems before they impact your real-time trading negatively.

To conclude, we’ve covered what backtesting in trading is, its purpose, benefits, and limitations. In the next section, we’ll delve into the difference between backtesting and other methods of analysis.

Different Types of Backtesting and Methods of Analysis

In the world of trading, one method has become increasingly pivotal to the success of traders – backtesting. This process involves testing a trading strategy on relevant historical data to ensure its feasibility before risking any capital. While backtesting has its critics, when used correctly, it can provide invaluable insights into market behavior and the performance of your strategy. In this article, we’ll explore the difference between backtesting and other methods of analysis and the importance of each in a comprehensive trading strategy.

Backtesting vs Chart Replay

Chart replay is a common technique used by many new technical traders. It involves going back in time on your charts, observing how your indicators move, identifying entries and exits, and recording this information in a journal or spreadsheet. While this can provide some insight into how a strategy may perform, it’s a time-consuming process and offers a very limited view of market conditions.

In contrast, backtesting using trading software allows you to input your trading rules and apply them over several years of market history across thousands of stocks or other financial instruments. The result is a far larger dataset, providing more information about the strategy’s performance across different market conditions and a larger number of trades. In short, backtesting gives you a wider and more detailed perspective, providing the comprehensive data necessary to refine your strategy.

Paper Trading vs. Backtesting

Another common analysis method is paper trading, a process where you follow your trading rules in real time, logging trades as they occur without risking actual capital. Paper trading has its merits, especially in terms of familiarizing yourself with the dynamics of the market and refining your trading process. However, similar to chart replay, paper trading lacks the comprehensive data necessary to assess a strategy’s profitability fully.

Different market conditions, such as bull markets, bear markets, and sideways markets, can drastically influence a strategy’s effectiveness. These cycles can last several years, making it impractical to gather sufficient paper trading data to assess a strategy’s performance across all market conditions. This is where backtesting’s ability to leverage decades of market data offers an invaluable advantage.

Single Instrument Backtesting vs Portfolio Backtesting

The choice between single instrument backtesting and portfolio backtesting often depends on the type of trader you are and the trading software you use. Some traders prefer focusing on one instrument at a time, an approach that suits single instrument backtesting. But for those who trade stocks or cryptocurrencies, where you might trade across a broad market universe, portfolio backtesting becomes critical.

In portfolio backtesting, you apply your rules to the entire market, picking out trending stocks according to your pre-set rules. Portfolio backtesting helps you assemble a portfolio of stocks or other instruments and evaluate its performance over time, a crucial factor for those trading in stocks or holding a portfolio of stocks.

Conclusion on Different Types of Backtesting and Analysis

While all types of analysis and backtesting methods offer unique insights, it’s clear that systematic backtesting provides a comprehensive understanding that no other method can match. Regardless of whether you’re using chart replay, paper trading, single instrument backtesting, or portfolio backtesting, it’s essential to remember that these are tools in your arsenal, each with their unique strengths and weaknesses.

As a trader, understanding these differences and applying the right tool at the right time is an important part of mastering the market. Backtesting, when done properly, not only equips you with an advantage over other traders but also instills a level of confidence that can’t be achieved through any other method of analysis.

In the next section, we will delve into backtesting different types of instruments, providing you with a comprehensive guide to optimize your trading strategies across various markets.

Backtesting Different Types of Instruments

In this section, I’m going to talk through each of the different asset classes and how backtesting in those asset classes differs. This will save you a lot of time and energy as well as help you avoid a bunch of mistakes that could really hold you back because backtesting each asset class is quite different.

Backtesting Stocks

So let’s first talk about backtesting in stocks since this is probably the most tricky to get right and once you understand backtesting in stocks, you can very easily apply that to many of the other asset classes.

So here’s the challenge in stocks…

There are literally thousands or tens of thousands of different stocks you can choose from. And so when you are backtesting a trading strategy in stocks, you need to have rules that sort through that huge volume of stocks and data to find the trades that you actually want to take.

Narrowing the universe of tradeable stocks using a liquidity filter

The first thing we need to do is filter out the stocks that are basically trash including:

• stocks that have no liquidity
• stocks that have extreme volatility
• stocks that don’t trade very often

Our trading strategies need rules to eliminate all of the garbage that we just don’t want to trade.

One of the best rules to do that is having a liquidity filter. For example, having a rule that requires a stock to turn over $500,000 – $1,000,000 per day on average eliminates most of the junk that’s listed on the market that’s not really tradable. You can trade stocks with lower liquidity than $500,000 turnover per day, but you’ve got to be careful because the liquidity starts to get patchy and your trades will become hard to get in and out of. The stocks also move erratically making your trading results unpredictable.

Taking Stock Market Cycles into account when backtesting stocks

The stock market goes through different phases which include long bull phases, bear markets and sideways markets. Most strategies will only work well in one or two of those different market conditions and typically won’t work across all market conditions. So in stocks, what we do is we have rules that isolate the market cycles that we are most interested in. This is called an index filter or a market regime filter.

For example, if you’re trading long side trend following, you may only want to do that when the broader stockmarket is going up. You generally won’t want to do long side trend trading when the market is in a strong bear market because there is a high chance that those trades will lose money even if the individual stock looks strong. We avoid these market conditions by adding a market regime filter rule to ensure we only take signals at the right time.

What trading rules do you need to backtest stock trading systems?

Once you’ve got the market regime right, then you get down to the trading rules to trade each individual stock. There are tens of thousands of stocks that we can trade on the US market alone, let alone all the other markets around the world (which you should be interested in trading by the way). I believe it is extremely important to diversify into different markets rather than just sticking to your home market (even if your home market is US Stocks).

So how do we find our buy and sell signals for each stock when backtesting?

Your trading system must include objective rules that tell it when to buy and when to sell a stock. These rules are used by your backtesting software to scan the historical data and find every single bar where there was a buy or every single bar where the sell conditions were true. The challenge now becomes that your buy and sell rules actually give you way more trades than you can possibly take.

For example if you trade a breakout trend following strategy then each day you could have hundreds of signals to choose from!

In fact, could be 10, 20, or a 100 times more signals than you could actually take. So the next thing you need is in stocks, is add portfolio construction rules to tell your backtesting software the order of preference to take the trades, how many trades to take and how many shares of each stock to buy / sell.

The position score rule gives your backtester the priority of each trade based on some sort of price behavior. Here are some example position scores for you to consider:

• Prioiritize stocks that have moved the furthest today compared to a 100 days ago
  Close / Ref(Close,-100)
• Prioritize stocks that have the lowest volatility
  ATR(14)/Close
• Prioritize stocks that have the highest volatility first
  Close / ATR(14)
• Prioritize stocks that have the furthest above its 200-day moving average
  Close / MA(C,200)

Those are all examples of position score rankings that you can use in order to prioritize which signals you should take. 

At this point I can almost hear you asking “which position score formula should I use?”… and the answer is of course to Backtest it!

So at this stage of the process we have got:

• our instrument liquidity rules to filter out the trash stocks we don’t want to trade
• our market regime filter to tell us whether we’re in a bull / bear / sideways market
• our buy and sell rules to generate the signals on each stock
• our position scoring rules to rank the trades so the backtest knows which order to take trades

The last step is to define a rule to calculate the position size. The position sizing rules (also known as money management rules) tell us how much capital to put into each trade, and you need to put those position sizing rules into your backtesting software so that it can assemble a portfolio and show you how that portfolio would’ve performed over time. Position sizing is one of the most important aspects of trading and it has a huge impact on your trading system’s performance.

All of those considerations come together to give you a complete stock backtest.

Survivorship Bias – 2 HUGE Challenges you must overcome to backtest stocks correctly

1. Index constituency survivorship bias
2. Stock delisting survivorship bias.

If you don’t have the historically accurate index constituency data for every single day of market history, then you can’t backtest or trade strategies that limit their universe to index constituents. You have to backtest the entire universe of stocks (all listed stocks) rather than the index constituents.

1. They have gone bankrupt
2. They have been acquired by another company

Backtesting Crypto

Backtesting strategies for cryptocurrencies comes with its own set of challenges, primarily due to the limited historical data available for most crypto assets. Unlike stocks with many decades of data, most cryptocurrencies have only a few years of historical trading data. Consequently, accurately evaluating the performance of trading strategies becomes more difficult.

Like stock backtesting, liquidity and market conditions are crucial considerations when backtesting crypto assets. Illiquid and highly volatile cryptocurrencies can be challenging to trade effectively, making focusing on tokens with more substantial liquidity and stability essential.

The primary challenge with crypto backtesting lies in acquiring sufficient historical data for robust evaluation.

Relying on data from a single exchange can lead to a skewed database, as the number of tokens available for analysis will be limited to those listed on that specific platform AND the length of history will be determined by the data the exchange listed that asset.

To address this limitation, it is advisable to use comprehensive datasets provided by reputable sources like Brave New Coin. These datasets amalgamate data from multiple exchanges, allowing traders to access a more extensive historical dataset with a broader range of tokens, leading to more comprehensive and accurate backtests.

For example, let’s say we want to backtest a trading strategy that involves Bitcoin and Ethereum. If we rely solely on data from a single exchange, we might have limited historical data for these tokens, as the exchange may not have listed them from their inception. By using a comprehensive dataset from a provider like Brave New Coin, we can access a more complete history of these tokens’ price movements, enabling a more accurate and reliable backtest.

Considering the dynamic nature of the cryptocurrency market, with new tokens continually emerging, it is crucial to obtain historical data that covers as much of the market’s history as possible. Access to a more comprehensive dataset enables traders to make well-informed decisions when developing and backtesting their trading strategies.

Aside from these crypto specific backtesting issues, all of the same considerations discussed in the stocks section also apply to backtesting crypto trading strategies.

Backtesting Futures

Backtesting futures is quite a different beast than stocks and crypto for two reasons: futures contracts expire and futures are a leveraged instrument.

You can’t just buy one futures contract and hold it for six months or a year because chances are it’s going to expire during that period. Because futures contracts expire and you need to roll from one contract to the next, backtesting with the individual contracts is problematic. So rather than using each individual contract, when you backtest futures you use continuous contracts.

Continuous contracts take all of the different expiry months and splice them together back adjusting as it goes to give you a continuous and smooth price series that mimics what your account would’ve done had you bought that contract, held it to near expiry and enrolled to the next contract and held that one, and then rolled to the next contract and held that one, and so on.

So you need to use those continuous contracts or have trading software that does the contract rolls for you. So that’s generally going to be more specific trading software specific to futures. So software like Trading Blox can do that. Software like AmiBroker, which is what I use, uses continuous contracts rather than managing each individual contract roll.

With futures, the other consideration is whether you are doing portfolio backtesting or single instrument backtesting. Most retail traders probably don’t have enough capital to do run portfolio futures trading strategies. You’ll need several hundred thousand dollars in your account to run a diversified trend following strategy in futures, for instance, because the margin requirements are quite high.

Over time, as the futures exchanges introduce more mini and micro contracts, the contract size becomes less and less of an issue, which is fantastic for traders, but right now there’s still a lot of contracts you can’t trade unless you have a quite large account.

So most futures traders, currently at least, have single instrument strategies because you can trade in and out on a single instrument basis without having a huge account. So if you’re trading single instrument, then obviously you’re going to backtest that way. It’s quite different to backtesting a portfolio of instruments like you do in stocks.

Backtesting futures is quite different to stocks, and you’ll want to make sure that you really learn and understand all of those considerations in detail before launching into futures trading. In particular you need to take into consideration:

1. Contract Rollover: As mentioned above, Futures contracts expire, so your backtesting (and live trading) needs to account for contract rolls at regular intervals. This introduces complexities when backtesting, as historical data must account for these rollovers, so at the very least some additional fees should be built in to account for rollover costs. In addition, the choice of rollover rule which determines when your position rolls from one month to the next can significantly impact results.
2. Leverage: Futures are traded on margin, meaning that traders can leverage their positions. The margin requirement is set by the exchange is is different for each futures contract. This leverage introduces an additional level of risk that must be accounted for during backtesting.
3. Volatility: Futures can be more volatile than stocks (some contracts more than others), especially commodities futures which can be influenced by a range of unpredictable factors such as weather, geopolitical events, or sudden changes in supply and demand.
4. Liquidity: Contrary to what you might think, some futures contracts can be quite illiquid, and liquidity changes as the time to contract expiration approaches. We need to be careful to ensure we are trading and backtesting the most liquid contract months and our positions roll forward to the next month before liquidity in our current contract dries up. Low liquidity can lead to slippage, which is the difference between the expected price of a trade and the price at which the trade is executed.
5. Price Limits: Futures markets often have daily price move limits which are set by the exchange in an attempt to ensure an orderly market and to protect the integrity of the exchange system. Price limits mean the price cannot move more than a predefined amount in a single day. If a price moves up (or down) buy the limit amount then it becomes ‘locked limit’ which means essentially that no further trading can happen that day. This can create gaps in the price series, affecting certain strategies, particularly those that rely on intra-day data.
6. Dividends and Corporate Actions: These exist in stocks, but do not occur in futures contracts.
7. Short Selling: While short selling stocks can be restricted or costly, going short on futures is generally straightforward and is a normal part of futures trading. It is just as easy to go short as it is to go long in futures and there are no restrictions on shorting like there are when shorting a stock. This opens up some different trading strategies for futures which might not be viable for stocks.

Backtesting Forex

Backtesting Forex is very similar to futures except the contracts don’t expire and the levels of available leverage are much higher in Forex. When you trade forex you are simultaneously buying one currency and selling another. To maintain this position you pay a financing cost which is determine by the difference between the interest rates of the two currencies.

Forex traders typically don’t trade a strategy on a diversified portfolio of currencies. Instead they usually trade single instrument strategies. This makes it very quick and easy to backtest. You don’t have contract rolls like you do in futures. You don’t have lots of garbage stocks to filter out like you do in stocks and crypto. So Forex is actually quite simple. You can have trading strategies in Forex that are just literally a couple of rules (since you don’t need all of the filters that you need in stocks and crypto).

Now, getting profitable and consistent, that’s a completely different story!

Here are some of the main differences between backtesting Forex compared to stocks:

1. Market Hours: Forex markets operate 24 hours a day, five days a week. The stock market however only trades for set market hours 5 days a week. This means that forex markets can react to news and events more rapidly than stock markets and there is only a ‘notional’ open and closing time on each daily bar. Many Forex traders trade short term intraday strategies on time frames less than 1 day since the definition of a ‘day’ makes less sense in a 24 hour market, whereas in stocks the open and close of the day are important reference points.
2. Leverage: Forex markets allow for much higher leverage than stocks (and futures). This means that a small change in the price can lead to a large change in the value of the investment. Many Forex traders blow up their accounts because of the high leverage causing wild swings in their account value. This leverage needs to be taken into account when backtesting Forex trading strategies, as the risk and reward profile can be very different compared to unleveraged instruments like stocks and crypto.
3. Currency Pairs: Forex trading involves the simultaneous buying of one currency and selling of another, while stock trading involves buying or selling a single stock. This creates some degree of correlation between many currency pairs because one of side of the pair may be common between multiple currency pairs (eg. AUDUSD, EURUSD, GBPUSD, CHFUSD are all influenced by the strength of the USD). This creates a different dynamic that needs to be accounted for in backtesting and results in portfolio trading strategies that work in stocks generating large drawdowns in the Forex markets.
4. Interest Rates: When trading and backtesting Forex strategies there is a cost to hold the position overnight which is determined by the difference in the interest rates between the two currencies in a forex pair. Many traders trade intraday to avoid this cost which is known as the rollover rate or swap rate. When backtesting Forex trading strategies this rollover rate must be taken into account as the cost can quickly add up and errode system profitability.
5. Trading Costs: In forex trading traders generally do not pay commission as the cost is built into the spread (the difference between the bid and ask prices), while stock trading might involve both a spread and a fixed commission. Additionally, forex trading costs (as measured by the spread) can vary depending on the time of day due to differences in liquidity depending on which financial centres are open during the day. When backtesting Forex strategies we must assume an average spread so that we don’t overestimate the profitability of our forex trading system.
6. Liquidity: Major forex pairs are some of the most liquid markets in the world and so slippage is generally not a problem, however on minor pairs spreads can be quite wide, so Forex traders need to take this into account when backtesting exotic forex pairs.
7. Short Selling: Short selling is more straightforward in forex trading compared to stocks. Like in Futures, shorting in forex is as easy as going long. There are no restrictions or additional costs other than the rollover costs and spreads described above.

So backtesting Forex is probably one of the easier markets to backtest because you typically use single instrument systems and there are no rollovers, we don’t need as many filters because we just select a liquid pair and trade it. However, I think Forex is one of the most competitive markets, and I’ve found certainly that it’s far easier to generate very profitable strategies and stocks and crypto compared to Forex.

Backtesting Options

Backtesting options is very difficult because there is a huge volume of data to deal with as a result of all the different stocks, calls / puts, different strike prices and different expirations. So it’s masses of data!

Backtesting options trading strategies is very problematic because of the huge quantities of data involved.

To assemble all of the data, pull it all into trading software and correctly backtest it all is a huge and very, very difficult task. And to put it bluntly, most trading software is just not up to the task. In fact, I’m yet to find an options backtesting software that I’m really happy with, and even if I did, the data is very expensive because you’ve got to get a huge volume of data from the exchanges.

So backtesting options is very problematic.

Unfortunately, a lot of options traders resort to paper trading and chart replays, and they’re not backtesting properly as we have discussed earlier in this article. The reason I don’t actively trade options is because I can’t effectively backtest the strategies over a long period of time.

The real challenge is that in options, a lot of strategies might look very good, but there’s a big tail risk that can really hurt you. Unless you backtest the options trading strategy over a long period of time you won’t understand the full implication and impact of this tail risk.

There are an increasing number of backtesting platforms online for options trading, and I’m hopeful that in the very near future, those will become competitive and perform as well as the the backtesting software that I use for stocks and crypto, but they’re just not there yet. This is why I’m not backtesting in options right now.

Summary of considerations for backtesting different instruments

Each instrument has a different set of considerations as you have seen. We have discussed the considerations for stocks, crypto, futures, Forex, and options.

I like my traders to start in stocks because it’s the most understandable and it’s the simplest to conceptualize. Once you know that you’ve got to filter out the garbage stocks, use a regime filter, have buy/sell rules, position score rules and position sizing, backtesting stocks is straight forward. It becomes really powerful because you can assemble that portfolio and see how it would’ve performed over time and get really great insights into your trading strategy.

Crypto works amazingly well and is a great way to diversify your stock portfolio. Futures is also great if you have enough capital to do it or if you trade single instrument systems. Forex is very easy to backtest, but very competitive and hard to win in long term. And options just doesn’t allow you to backtest effectively with current technology.

In the next section I am going to discuss how to set up a trading strategy for backtesting. 

Setting Up a Trading Strategy to Backtest

In this section I will share how you can set up your trading strategy to backtest so that you can get an accurate picture of how your trading strategy performs. This process is important no matter what trading platform you use, so I have generalised the process to be useful for you no matter which backtesting platform or backtesting software you use.

Choosing a Time frame & Documenting Trading Rules

The first, and the most important step is to choose your timeframe and document your trading rules that you want to backtest. The time period could be daily, weekly, monthly, or it could be intraday. This doesn’t impact how you backtest provided you have the historical data at the right level of granularity. 

Documenting your trading rules is critical to do BEFORE you start the coding process. A lot of people will crack open the trading software and start coding madly, but this is a mistake. Before you do that, first document your trading rules in terms of pseudocode so you can check the  trading logic is complete.

Pseudocode is a series of logic statements written in plain English which define exactly how the trading system works. This is where you will document the various technical analysis rules and technical indicators you use for your entry and exit rules. 

If you do not have your own trading ideas then at this point you should look at The Trader Success System because it comes with a large number of complete trading systems ready for you to test and apply to your own trading account. Reading a large number of trading books will also help you generate new trading system ideas.

Ensuring your rules are 100% objective

Once you’ve documented your trading strategy rules, the next step is to ensure those rules are 100% objective. One big problem that new systematic traders face when trying to codify their strategy is eliminating the subjectivity from their trading rules.

Traders who use technical analysis often use checklists which can be notoriously subjective and impossible to backtest. All discretion must be removed from the strategy to allow backtesting!

If you can’t point to an indicator (or set of indicators) on a chart and say, this is exactly the conditions that tell me that buy / sell rule is true, then you have subjectivity in your trading rules. This must be eliminated before you can backtest the strategy.

Sometimes you just need to put a stake in the ground to give yourself a place to start:

• “The Stock is trending up” >>> Today’s closing price is above the 200 day simple moving average
• “The stock is not too volatile” >>> The Average True Range is less than 5% of the closing price

If you have got subjective rules, make them objective so that you can then code them and then backtest them correctly.

Fully Define Your Trading Strategy

Once your rules are 100% objective, you must ensure your strategy is fully defined. A fully defined trading strategy documents exactly what you would do under ALL conditions. So there can’t be any edge cases or situations where you have to intervene and manually make a subjective decision – there must be a rule for everything, else you cannot backtest it correctly.

You can only backtest objective trading rules, so think through every single circumstance, and make sure that the rules cover it. Here are some questions to help you work through this:

• How will you prioritize all the different buy signals?
• How many buy signals will take in any one day?
• When should the system stop taking buy signals?
• How many open positions will you hold at any one time?
• How will you calculate the position size for each trade?
• Where’s the stop loss? Is it in market / next day / next bar on open?
• Where’s the trailing stop? Is it in market / next bar on open?
• Is there a profit target? Is it in market / next bar on open?
• What other conditions under which will I exit?
• Is there any panic exit?
• If the index starts collapsing, is there an index exit?
• And so on…

Make sure every decision you have to make for that trading strategy is fully documented and codified. Work through all of the different scenarios, and make sure your rules cover all of them. Once you’ve fully defined your strategy, it’s time to code it.

Code and Error Check Trading Strategy Logic

Now is the time to (finally) open up your trading software and convert your pseudo code rules into code. If you’re a member of The Trader Success System, you get a code library and trading system templates as well as a whole portfolio of complete trading strategies. This means system coding becomes a simple copy and paste job into a preformed trading system template.

So it is It’s dead easy for members of The Trader Success System to codify their own strategies and backtest them. If you’re not a member of the Trader Success System, click the link to join, or, you’re going to have to learn how to code and master that yourself.

The best approach is to convert your pseudocode to code one rule at a time, then make sure you rigorously error check that code as you go. It is far easier to debug as you go rather than trying to get everything right in one go and debug everything at once.

Most trading software like Amibroker has code testing, syntax testing functionality. So you can press the AFL syntax verifier and it’ll tell you if you’ve got any syntax errors. But a syntax error is not the only error you’ve got to look out for. You’ve also got to look out for logic errors and make sure that the code is executing correctly as we discuss below.

Check Backtest Executes Correctly

Once you’ve fully codified your strategy and run your first backtest, you then need to look at several examples of each different type of trade your system generates to ensure they are all executing correctly. For example you need to look at the entry date /  price and the exit date / price for all of the following trade types to ensure on the charts that the backtest is behaving exactly as it should:

• Trades that hit their stop loss
• Trades that exit based on an indicator exit rule
• Trades that hit a trailing stop
• Trades that hit a profit target
• Trades that hit a time stop
• Plus any other combinations that your system generates

You need to check several examples of each so that you can be sure that the signal to buy and sell was generated on the correct bar, and then that was executed on the correct bar at the correct price. You do not need to check every single trade in the database, but you do need to check several examples of every type of trade outcome to ensure they are all executing correctly.

Once you’ve done that, you are now ready to backtest your trading strategy and really start to investigate performance. But you’ve got to do all of this work first and really make sure it’s coded correctly, otherwise any performance analysis you do will be flawed.

Spending time getting this right up front will improve your confidence and eliminate time wasted finding mistakes late in the backtesting process. Once you have done all of this you can go on to optimization, performance improvement, trying different ideas and fine-tuning your trading strategy.

You’ve done all that, it’s time to move on and backtest and optimize your trading strategy, to try and improve the performance so that you can make the most profit possible out of your trading.

Common Backtesting Mistakes

In this section I want to share six common mistakes traders make when backtesting their strategies. Trading mistakes have the potential to lose you huge amounts of money. So if you don’t avoid these then you’re going to have problems in your live account. 

Backtesting Mistake 1: Ignoring Slippage and Commissions

I can’t tell you how many times I’ve looked at backtest reports from signal services, from system vendors, from website selling courses that don’t include slippage and commissions in their backtest results. This is just criminal because it’s basically outright deceptive.

Publishing backtest results that don’t include slippage and commissions is deceptive and highly misleading!

These costs should be factored into your backtests because slippage and commissions are a real cost of trading and you cannot trade without them. Yes there are some brokers in some markets let you trade commission free, but they don’t allow you to trade slippage free!

There’s always a spread and you need to take into account the fact that you may not always get the price that you really want. Slippage and commission are extremely important and can ve very high costs compared to your systems net profit, especially for short term trading strategies.

What is the impact if you don’t include slippage and commission in your backtest results?

For a short term high frequency trading system that has a very fine edge, the impact can be catastrophic. If you have a mean reversion system, for example, and the average profit per trade is 0.5% and you get 0.25% of slippage on the entry and the exit on average, then your system is not profitable. It’s break even at best!

The impact of slippage and commission is most significant for short term, small edge or low expectancy strategies. For example strategies like mean reversion, where you’re trading many, many signals and they’re very short term moves. So you get a small positive gain when you win, and hopefully a small loss when you lose. Slippage and commission can really destroy the profitability of those systems if you are not careful or if you trade illiquid stocks.

If you’re looking at a system to buy or to get signals from and they don’t adequately account for slippage and commission, then you should run the other way!

Slippage and commission is a real and significant cost of trading, even though commission nowadays is very small. When I started trading, I was paying $40 to buy and $40 to sell. It was unbelievable, so much harder to make money, but now you can get a couple of bucks or a 0.08% commission and there’s even commission free.

But what happens if you ignore slippage and commission?

Ignoring slippage and commission makes short term, low edge strategies look relatively much more attractive. As a result you will be drawn to strategies which are shorter term, higher frequency, small profit per trade, and if you trade those with real money, then you will incur some slippage and commission and the profitability will very quickly fall apart and you will lose money!

With longer term systems like a long-term trend following system, it doesn’t make as much of a difference and is less likely to destroy the system.

How much slippage and commission should you allow for?

This depends on what markets you’re trading, how liquid they are and what commission you have to pay your broker. The first step is to check with your broker and see what commission you’re actually going to have to pay for each trade. Then look at the markets you’re trading and note how much slippage on average you’re actually getting.

To measure slippage, enter a market order to buy the stock before the market opens, when the market opens, look at the price you get, compare it to the official opening price, that is the slippage that you incurred. Monitor the slippage on a few trades, maybe 10 or 20 trades, and see what allowance you should have and then put that into your backtest. At the very least, 0.25% or maybe even 0.5% slippage and commission is a good rule of thumb to start with. 

Backtesting Mistake 2: Allowing Survivorship Bias

Mistake number two is allowing survivorship bias into your backtest results. As discussed previously, there are types of survivorship bias:

1. Index constituency
2. Stock listing or de-listing survivorship bias

Index constituency survivorship bias is where you backtest your strategy on the constituents of a certain index. So let’s say you want to trade stocks at the S&P 500, so you get the 500 stocks that are currently in the S&P 500 list and you backtest your strategy on those stocks over the last 30 years to see how well it would’ve performed.

This sounds like a great idea, but the trouble is the strategy is cheating because 30 years ago you didn’t know that 2023 those 500 stocks would be in the S&P 500 index. They could have been small caps way back then, but they’re in the index now, which means they probably went through a meteoric rise in share price and you wouldn’t have known that in advance if you were trading 30 years ago.

The way you avoid this is to have historically accurate index constituents. So every single day in the backtest, every bar in the backtest, the backtesting software looks at that stock and says, is that stock in the S&P 500 index today? If it is, the system can buy that stock, but if it’s not, it can’t buy that stock.

The only way that I know of to get historically accurate index constituent data is through Norgate data. I

The second type of survivorship bias is stock listing or de-listing survivorship bias. This is where you are backtesting stocks that are currently listed on the market today. The problem with this is you are ignoring a large number of stocks that existed in the past that have since disappeared.

There are several reasons why a stock might have disappeared. Maybe it went bankrupt and the price went to zero (the worst case scenario) and more commonly, the stock gets acquired by another company.

If a stock gets acquired it is typically at a premium to the share price at the time of acquisition. If you are holding a stock when an acquisition is announced you therefore get a boost in your trade profits before the stock delists.

It is fashionable (in trading guru educator circles) to claim that you need all the de-listed stock data to properly evaluate your trading strategy and so that you don’t delude yourself. But while it’s better to have delisted data than not because you’re getting a complete view, what I’ve found is that actually including the delisted stocks in my backtest generally makes my systems better, not worse as many fear mongering gurus claim.

If you design a system that will only hold a stock on the long side, if it’s going up and has some exit, like a trailing stop or a stop-loss that prevents you from holding a stock that is going down, down, down, down, down, then delisting due to bankruptcy is not really an issue because you’re going to get an exit signal and get out well before the stock goes bankrupt.

Very rarely will a stock be trending up strongly and then disappear in a puff of smoke, bankrupt. In 25 years of trading, and after tens of thousands of trades, it has only happened to me twice! Plus it has never happened to me on Australian, US or Canadian stocks (only on Shanghai and Hong Kong stocks). I am certainly not saying that it can’t happen, but it is certainly highly unlikely as my experience has shown.

Backtesting Mistake 3: Allowing Look-Ahead Bias (Future Leaks)

The third common mistake is allowing look ahead bias or future leaks into your backtest. This is where your backtest rules somehow ‘peak’ into the future at information that you wouldn’t know at the time of the signal.

A subtle but common example of this is having a trading strategy that enters at the close of the market. It enters market on close, but in order to calculate the signal to buy the stock at the close, you need the closing price.

You cannot always execute this type of strategy faithfully because it cheats a little bit. It needs information that’s available right now to place a trade right now, and that can’t generally be done in real time trading (even though your backtest thinks it can!)

A more serious example is having an indicator that looks forward into the future (the Zig Zag indicator is an insidious culprit here – NEVER use the Zig Zag indicator!).

Some common mistakes in this area include:

• referencing tomorrow’s opening price instead of yesterday’s
• referencing the moving average on the day you enter at the open (hint: you can’t calculate most indicators until after the close)
• Using today’s average true range to set your in market stop-loss level today (you have to use yesterday’s ATR, not today’s). This is very subtle because let’s say you enter your trade at the open today, and you want to place your stop-loss. In the backtest, you could code your system to set that stop-loss as five times the average true range. The trouble is at the open when you’re setting that stop-loss, you don’t yet know what today’s average true range is. You only know you yesterday’s average true range. So by using today’s average true range, you’re cheating. Your system is looking into the future and thereby rendering your backtest incorrect and faulty.

If you use future information to make a decision in your trading now, it inevitably looks fantastic in the backtest but you obviously can’t do it in real time trading. So as soon as you get a backtest result that is a beautiful smooth line from bottom left to top right, the equity curve is really smooth, low volatility, chances are you’ve got look ahead bias. You’ve got a future leak in your system code, so you have to avoid this at all costs.

How do you avoid future leaks in your backtesting?

The way to avoid future leaks in your data is to look at every rule and say, okay, at the time of the trade, would I know that information? Can I see that on the chart when I’m supposed to place the trade? If you can’t yet see that information on the chart at the time you’re supposed to be placed in a trade, then it’s not yet available and it’s a future leak.

Future leaks are dangerous because your backtest to be overly optimistic, which will cause you to be too aggressive and you actually won’t be able to make money in real time trading. So look ahead bias or future leak is one of the biggest downfalls of backtesting if you get it wrong.

Backtesting Mistake 4: Information Availability Timing

Information availability timing problems are like a future leak but a little more subtle because in your historical data, they may not show up as a problem at all. Let’s say, for instance, you are using a trading system, and you incorporate some external data like the rate of inflation or the current interest rates. You can do this if you have the historical data source and you are using backtesting software like Amibroker.

If you pull in the foreign data source into your backtesting software you can certainly backtest it and see that it helps your signals. The problem comes when you switch to real-time trading and discover that some external data like fundamental information is released late and then the historical data stream may be giving you information that will not be available to you in real time because the release is delayed and the information is backdated once it is released.

You need to know the release schedule and line up the information you have available at the time of the technical signal from your system with the information available from the fundamental data source you’re using. If the fundamental data takes several weeks or months to come out and be updated, then you’ve got to ensure that you’re only using what is currently available at the time of the signal. This is important because it’s just like a future leak – your backtest will look great, but you can’t actually generate the signals in real-time trading.

Execution Timing Mistakes

Execution timing mistakes in your backtesting are where your backtest assumes that the trade gets placed at a certain time and gets filled, but in reality, you couldn’t do what your backtest is assuming. For example if your stock trading system generates a signal at the close, but the backtest is entering at the open on the same day. This is another form of future leak because you just can’t do that in the real world.

Future leaks and execution timing mistakes in your backtest will give you hugely optimistic results that can’t be replicated in real life trading.

As soon as you see hugely optimistic results in the form of very smooth equity curves or high rates of return with low drawdown, that needs to be your cue to look for future leaks and other mistakes in your trading system code.

You must check the logic and make sure you get the data to generate the signal and then place the trade, and you can do those steps in sequence in real life trading. If unsure, then try and paper trade your system and see if it works. If you’ve got all the information you need, great. If you don’t yet have the information you need at the time you’ve got to place the trade, that’s your clue to tell you that you’ve got something wrong. 

Backtesting Mistake 6: Misusing In / Out of Sample Data

Misusing your data when you are backtesting is a common problem that causes many systematic traders to lose money. It is tempting to throw all of your data into the backtester and optimize your strategy over the entire set of data and generate the best system you possibly can. The problem is, if you do that, you don’t have any data left to validate your strategy.

If you don’t save some of your data for out-of-sample testing how do you then validate that you didn’t curve fit it?

If you burn all your data by backtesting and optimizing on the entire data series, the only way to validate the system is to wait 6-12 months and incubate your strategy on that new data to ensure that it works on the unseen data that was not used in your trading system development. The trouble with incubation as a method of validation like this is that it wastes a ton of time.

A better way is to take your original data set and segregate it into some sort of in-sample and out-of-sample groups. Maybe you use half of the stocks and half to two-thirds of the history for the design optimization and use the rest of the data for validation. There are lots of different mechanisms and approaches to doing this, but the key point is you must have some out-of-sample data to validate your strategy on. If you don’t, then you need to incubate it for long enough to be sure that you didn’t curve fit it, and you don’t have any fault in your backtest.

Data Requirements for Backtesting

In this section we will cover the data requirements for backtesting. First we will cover how to get the right historical price data. Then we will cover some of the data problems and challenges that you can have when choosing data, adjusting data for correct backtesting, and recommend data sources for you to use.

Getting Historical Price Data

When you are backtesting stocks and developing stock trading systems you need a complete data set for all stocks listed on the exchange with as much data history as possible. This data should ideally include currently listed stocks as well as delisted stocks.

You can get data history for the currently listed stocks for free from sources like Yahoo, but the trouble is when you get free data, there are several problems and risks. The data is the most important input into your systematic trading decisions, you want the data that you are using to be high quality because if you’ve got low quality data with mistakes, errors, and omissions, then you can get faulty trading signals.

Faulty trading signals due to poor quality data can lead to losses and drawdowns in your account. So we want to avoid that at all costs. So my guidance is to avoid free data. It’s just not worth it. 

You should get your data from a reputable data provider such as Norgate Data. Good data vendors get the data from the exchange, clean it, make sure that all of the price bars are correct and there are no bad ticks. They will also make sure all of the data is split-adjusted and check for errors and omissions before serving it up to you.

The other advantage of using a good end-of-day data provider is that they usually come with a downloading software or application to make the download quick and easy. The two data vendors that I use have really simple, easy data downloaders that can be automated so that your data is up to date whenever you open up your trading software

The key message for your data history is you need to pay for it. Don’t skimp or you will end up losing money in your real-time trading.

Data problems in Backtesting

There are 3 data problems that you must be aware of:

1. Bad ticks
2. Data not adjusted for splits
3. Data not adjusted for dividends

Bad ticks in your data history

If you get free data and you look very closely at enough of it, what you’ll notice is that there’s some bad ticks in the data. Typically the way this manifests is the high, the low, the open, or the close is wrong for a particular bar.

For example is, let’s say, the stock price is $10.50 and the open and the high and the close are all around $10.50, but the low might be all the way down at 10 cents or 10.50 cents, and that’s a bad tick. It’s data that hasn’t come through clean from the exchange to the data vendor and then to you. In this example the low should have been $10.50, but instead it was 10.50 cents. If you get cheap or free data, you will find these bad ticks where they’ve got the decimal point in the wrong place or maybe either the open or the high or the low or the close is set to zero or something like that.

This happens more regularly than you would like and can cause a significant problem for your backtesting and signal generation.The problem because you want to be able to have confidence that your backtest results are valid so that you’ve got confidence in your system.

But any data errors can cause faulty results. You might end up getting outliers in your backtest, really big winners or really big losers that maybe are not realistic trades. So make sure that you get good quality data free of bad ticks… and then in your backtests still check regularly for outliers (both positive and negative) to ensure there are no anomalous results caused by data problems.

Adjusting data for stock splits

The second problem is data that has not been correctly or consistently adjusted for stock splits (or reverse stock splits). Most free sources don’t split adjust their stock data history.

A stock split is when maybe a company divides their shares into a larger number of smaller shares to reduce the price of each share. If a stock is trading at high price per share, and they do a 10 for one stop split, the number of shares on issue increased by 10x but the price of each share drops to 1/10 what is was previously.

For example, if a stock is trading at $100 per share and you are holding 1 share, then tomorrow they do a 10-for-one split. This means you will then be holding 10 shares at $10 instead of one share at $100. Your position is still worth exactly the same amount of money ($100), but the price of the shares has dropped dramatically. If you want help calculating the impact of stock splits, check out my trading calculators resource page.

If the data provided that you are using doesn’t split adjust the data, the way that’ll look on the chart is the stock’s up at $100 a share and then the next day it drops down to $10 a share, and there will be a big big gap down. Now, clearly that gap down is not a failing of the company. The company is not going bankrupt or anything, they just had a stock split.

What the data vendors should be doing is saying, “Okay, well, today it’s $10 a share. Yesterday was $100 a share. What we do is from yesterday back in time, all of those share prices get divided by 10 to ratio them down to match the current price post split.” That way you get a nice, smooth, consistent price series, and when you split adjust correctly, the performance of the chart split adjusted matches what you would have if you bought and hold that stock.

So it is important to have split adjusted data, otherwise, you have these big gaps down in the case of a stock split, or a big gap up in the case of a reverse stock split. This will give you faulty signals and ruin your backtest.

Adjusting data for dividends

It is also ideal if your data is adjusted for dividends and capital returns / capital reconstructions. This is important because if a stock declares a dividend, the price usually drops by the value of the dividend, this can cause you to get incorrect signals from your trading system if the data is not dividend adjusted.

Again, let’s say the stock’s trading at $10 and it declares a 50 cent dividend. On the ex-dividend date, the stock price will typically drop by the price of the dividend, say 50 cents, and then continue trading. That looks like a little gap down, and that looks like a negative sign on the stock chart and often that can trigger your exit signals or stop losses. But it’s a dividend, not a drop in the share price, so it should not cause you to get trading signals on that stock at all.

If you get dividend data with your stock history, then you can have your trading software adjust the historical price of the stocks for all of the dividend payouts, which reduces those little gaps caused by dividend payments.

Unfortunately not all markets are easy to get dividend data for, but if you trade Australia, US, or Canada, you can get dividend adjusted data from Norgate Data. For most other markets, you can’t really get it so you’ve got to just put up with these small dividend related gaps.

Not having dividend adjustments are not the end of the world because it doesn’t completely destroy your backtest, like stock splits can. But if you can get dividend adjusted data, it is better.

For capital returns and reconstructions, it’s even more important because they tend to be much bigger than dividends relative to the stock price. So you want to have split adjusted, capital adjusted, and dividend adjusted data if you can.

The other adjustment that sometimes you want to use is padding of the data. Not every stock trades every single day. So for example, if you trade small cap mining stocks, there will often be trading halts when news is announced to the market. They have to go on trading halts so they can announce the news. Everyone gets all the information at the same time so it’s a fair and equitable market, and that means that there’ll be some price bars where there’s just no movement. So what I do for a lot of my systems is pad the data so that the price of the individual stock lines up to the price of the index.

The reason you want to do that is so that you get accurate index signals in your backtest. If you’re using a market regime filter based on your index, then adding that padding to each of the stocks to make sure that each of the price bars line up between the stock and the index, even if the stock was in trading halt means you get accurate market regime filter signals, and that’s really important, to turn your system on and off at the right time. So they’re the ways you want to adjust your data.

Recommended data sources for backtesting stocks

So now that you know what you’re going to do with your data, what are my recommended data sources for backtesting?

Well, my number one recommended source for end of end-of-day data for backtesting is Norgate Data.

The reason I really love Norgate is they have super good quality data. I have to the team at Norgate and they go to great lengths to clean and make sure their data is high quality, make sure all of the split adjusting is done. They also provide historically accurate index constituents and delisted stock history. To top it all off, the Norgate Data downloader is fantastic. It does all of the maintenance quickly and easily and it can be automated, so it downloads the data whenever there is an update available.

The ONLY problem with Norgate Data is that you can only get US stocks, Canadian stocks, and Australian stocks plus futures and Forex data.

If you want to trade any other markets other than those, you can’t use Norgate. So the data provider that I recommend for any other markets is Metastock end-of-day data. Metastock end-of-day data covers a broad range, but pretty much all stock markets globally for a very, very reasonable price. Metastock also have a data downloader that can be automated by scheduling the download task in the Windows task manager to run at the same time every day.

Depending on which markets you want to trade, you’ll subscribe to either the Asian region, the European region, or the North American region, and for one low price of about $30 a month, you get access to all markets in that region. Metastock is a really great source if you want to trade some of the less mainstream markets. I use Metastock data for the Hong Kong market, Japan, and the European markets as well.

The data comes in Metastock format, the industry standard format for data, which can be used in pretty much any trading software package. So they’re the data requirements for backtesting.

>You can get a one-month free trial to Metastock end-of-day data with this link.

Optimization in Backtesting

What’s the first thing you want to do when you crack open your backtesting software and you put in your trading system?

I bet you it’s optimization because everyone wants to optimize their trading system to maximize the profit that you can make from your trading rules. But is optimization really what you should be focusing on? In this section I will introduce you to three styles of optimization: individual parameter optimization, brute force, and walk forward. But first of all, what is the purpose of optimization and why do we want to do it?

Well, I’ve written a whole blog post on how optimization can make or break your trading system, but let me just summarize it for you…

Optimization is about maximizing your probability of future profit. It’s not about maximizing your backtested profit. Because when we optimize and backtest, we’re working on past data. And if we optimize to find the most profitable parameter combination, what we end up doing is curve fitting and cherry-picking the very best combination. In the past, on your backtest that’s going to look great – you will get an amazing equity curve that’s nice and smooth. However, in future real-time trading, that system is going to fall apart because you’ve curve fit to the very best parameter values.

Instead of focusing on the maximum profit that you can get from your past backtest, you should instead focus on maximizing your chances of future profitability.

So how do we do that?

Well, we run optimizations to find areas of stability within the parameter space. If your system is profitable with a 50-day moving average, but it loses money with a 45-day moving average and it loses money with a 55-day moving average, it’s not a very stable system. But if you run an optimization, you find that with a 70-day moving average, the system is profitable but not quite as profitable as maybe it was at 50, but at 60 it’s profitable, and at 80 it’s profitable and it’s 55, it’s profitable, and at 85 it’s profitable. This tells you there is a nice stable area of good performance around 50-60. That’s a much more powerful discovery because that broad area of strong performance is likely to give you profitable future trading. If the market shifts a little bit, behavior shifts in the future, which it will, then that parameter value is more likely to be profitable.

Whereas compare that to the profit peak that you found when you just did an optimization and chose the very best combination, that profit peak was unstable, and therefore, in future trading, probably not going to actually be profitable.

In backtesting, in optimization rather, the first thing you want to do is get your focus, your mindset right. We’re looking at maximizing probability of future profit, not maximizing future profit dollars.

Okay, let’s talk about the three styles of optimization right now: individual parameter, brute force, and walk forward. Individual parameter optimization is first.

Individual Parameter Optimization

Individual parameter optimization is when you code your system and take each parameter one at a time and you vary it from some low number to some high number, and you choose the value for that parameter that gives you the performance that you’re most happy with.

As I said earlier, the parameter value you select should be selected based on a broad area of stable performance rather than a narrow spike of very high performance. We want stability and profit, not just maximum profit.

Individual parameter optimization is you take them one at a time. You vary it, you choose a value, and you input that into the system. Then you move on to the next parameter. You vary it, you choose a value and you put that into the system, and so on and so on down through all of the parameters in the system.

The great things about individual parameter optimization are:

1. It is easy on the computer. It doesn’t take a whole lot of computing power, so it’s pretty quick.
2. It is easy to visualize. If you’re a visual person, you like to see the data when you’re just optimizing one parameter a time, you get a two-dimensional chart, XY sort of access, and you can see the profitability as you vary the value of that parameter. And you can, using that visual illustration, choose the value that you’re most interested in.

If you try and optimize three, four, or five variables or parameters simultaneously, you can no longer visualize that performance because there are too many dimensions, so it becomes much harder.

Individual parameter optimization is quick, it’s easy, and it’s a visual. That’s a really great set of advantages. If you’re a beginner trader and you’re just starting to optimize trading systems, optimizing parameters individually is by far the best way to do it.

Brute Force Optimization

The next approach involves the brute force optimization method, an approach that many traders enter the market with and immediately implement. For instance, they may choose a triple moving average system as their trading rules. This system has three parameters – one for each moving average. They assign a wide range to each parameter and optimize all of them simultaneously. After testing 365,000 combinations, they select the top-performing one and begin trading. However, they often encounter losses in real-time trading.

The primary reason for these losses lies in the fact that they’ve excessively tailored their trading system to past data. Testing all possible combinations and selecting the best among them may seem effective, but this approach doesn’t guarantee future performance.

Brute force optimization presents a significant pitfall. Trading software often encourages this method by showcasing their speed and power. They entice users to input all parameters, optimize them, and select the top-performing one. Moreover, these software offer sophisticated optimization functions like particle swarm, which sounds complex but essentially involves optimizing a smaller data subset. Despite starting with a few combinations, they still aim to identify the best combination among all parameter combinations in the optimization. This leads to curve fitting, a common problem.

To effectively implement brute force optimization, you need to be strategic. I teach an innovative method in The Trader Success System known as robust brute force optimization. Developed to enhance my trading and optimizations, I teach this to all my students as it allows you to identify areas of stability within the multi-dimensional parameter set space.

The process is straightforward. Set up your entire optimization to run on brute force and let your computer run for a couple of hours or overnight. You’ll end up with thousands of data rows. Then, use my method to manipulate this data and pinpoint areas of stability. This provides you with the correct parameter values that are likely to be successful in real-time trading.

I’ve assessed the effectiveness of this strategy using walk forward techniques, comparing standard individual parameter optimization against my robust brute force optimization. The latter is a proprietary process only taught to members of The Trader Success System. By running this robust method and projecting it several years into the future, I’ve found that the robust brute force optimization method consistently provides the most stable and reliable way to optimize a trading system.

Therefore, while conventional brute force optimization poses risks, using the robust brute force optimization approach I teach in The Trader Success System can be a powerful addition to your systematic trading toolbox.

Walkforward Optimization

The third optimization method I want to discuss is walk forward optimization. This technique was developed and promoted by Robert Pardo, a pioneer in the systematic trading realm. Walk forward optimization takes a segment of the data, optimizes the system and all its parameters, selects the best parameter value based on that data slice, and then projects it forward, testing it on an out-of-sample set. This process is repeated for each subsequent time period. The output from your walk forward optimization is the pieced-together out-of-sample backtest from all these steps.

Walk forward optimization is often considered the gold standard in optimization, and I agree, but only under certain circumstances. It excels in the context of short-term systems, high-frequency systems, and systems with a fairly standardized, narrow distribution of trade outcomes. In each optimization window (three months, six months, 12 months, 24 months, etc.), you get a multitude of fully developed trades that the optimization can base its decision on.

However, this method is less effective for longer-term systems like long-term trend following, characterized by numerous small, short-term losing trades and a handful of large winning trades that can extend over several months.

If you try walk forward optimization for a long-term trend following system, the losing trades are fully developed in the optimization window, but the winning trades are not. These winning trades are truncated, cut off when the optimization ends, leading to suboptimal optimization decisions, as too many of the trades, especially the winning ones, are truncated.

Therefore, I’ve found walk forward optimization to be highly effective for short-term, high-frequency, high-turnover systems but less so for long-term, slower-moving systems like trend following.

It’s crucial to choose the appropriate optimization method for your systems. For long-term trend following and medium-term stock trading systems, I recommend my robust brute force optimization method. Alternatively, you may prefer the individual parameter optimization or careful brute force optimization, ensuring you seek stability. For frequent, short-term systems such as mean reversion, walk forward optimization is typically the best choice.

In conclusion, when it comes to optimization and backtesting, there are three primary choices: individual parameter optimization, brute force optimization, and walk forward optimization. There’s no universal winner; it’s crucial to select the right tool for the job.

In this discussion about optimization in backtesting, we’ve covered these three methods and even offered a glimpse into the robust brute force optimization method exclusive to members of The Trader Success System.

Backtesting Software Review

Making the right choice of backtesting software can be overwhelming, given the range of features and functionalities on offer. The right backtesting tool will make your trading system development substantially easier and faster. Therefore, in this section, we aim to provide an overview of common backtesting software, discuss their advantages and disadvantages, and offer some recommendations on the most suitable backtesting software for different user needs. Be careful – just because one package is a common backtesting software does not mean it is up to the task as you will soon see!

Amibroker Backtesting

First on our list is the Amibroker backtesting program. Despite the broker-like name, Amibroker is a highly potent trading software for portfolio-style backtesting. This means it can backtest a system that may concurrently hold up to 20 or 30 different stocks in a portfolio, managed by the system rules for entry, exit, and position sizing.

Notably, Amibroker is very affordable and fast, making it a top choice for those starting. It also offers a fully-featured free trial for potential users to understand its functionality. It’s worth noting that the software uses its own coding language, Amibroker Formula Language (AFL), which is moderately easy to learn, somewhat akin to writing complex Excel formulas.

However, Amibroker does have some limitations. Certain functions require the use of complex code in a ‘custom backtest’, which may be off-putting for some users. Also, while Amibroker has a large user base, its community can sometimes be less than friendly. On the bright side, if you want to learn Amibroker and need backtesting support, the Trader Success system offers a positive and supportive community. The only significant drawback of Amibroker is its inability to test multiple systems simultaneously in a portfolio.

Realtest Backtesting

Next up is RealTest, a relatively new but brilliant piece of backtesting software created by Marsden Parker, a noted market wizard. Its coding language is intuitive and easy, and it easily handles limit order systems and can test multiple system portfolios simultaneously. RealTest even offers integration with Interactive Brokers to manage order placements, a feature not native to Amibroker. Though it is an excellent piece of software, it’s important to note that the community of RealTest users is still growing, and its dependency on its founder is high.

Tradestation Backtesting

TradeStation is a well-established name in the industry, combining a broker service with a backtesting platform. This unique setup is convenient as the broker feeds data directly into TradeStation’s software for analysis and backtesting. While this is a big advantage, the software is old and slow, in need of a revamp. The backtesting is painfully slow compared to Amibroker and RealTest, making it unsuitable for those needing to test portfolio systems.

Multicharts Backtesting

MultiCharts uses the same language as TradeStation, EasyLanguage, but with better backtesting capabilities. It can perform portfolio backtesting and also has built-in automation capabilities. While it isn’t as fast as RealTest or Amibroker, MultiCharts is a good choice for those wanting to use EasyLanguage for their backtesting.

Metastock Backtesting

Metastock is a stalwart in the industry and an excellent option for discretionary chart-based traders due to its robust technical analysis tools. However, its backtesting is limited to single instrument systems and lacks the sophistication seen in Amibroker or RealTest. Despite this, Metastock boasts a supportive community and a plethora of third-party coders and additional add-on systems and indicators.

Tradingview Backtesting

TradingView is a fantastic platform for discretionary technical analysis with a great charting interface. However, its backtesting functionality is rudimentary at best. Backtesting requires the use of Pine Script, which is more difficult to learn compared to EasyLanguage or AFL. Moreover, it only supports single instrument systems. For systematic trading, it’s better to opt for software like Amibroker, RealTest, or MultiCharts.

Excel Backtesting

Excel is often asked about for backtesting, and while it can be used for this purpose, it’s not ideal. Excel lacks the built-in backtesting protocols, reports, and output charts seen in Amibroker or RealTest. It may be a free option, but using software designed specifically for backtesting is a more optimal choice.

Interactive Brokers Backtesting

Finally I wanted to touch on Interactive Brokers backtesting. I see a lot of people online asking, how do I backtest with Interactive Brokers? And the short answer is you can’t. There is no native backtesting capability in Interactive Brokers’ platform. If you want to backtest your trading systems, then you need a separate piece of software that interfaces with Interactive Brokers.

If you use Interactive Brokers and Amibroker, at Enlightened Stock Trading we’ve developed The Smart Stock Automation Engine, which links Amibroker and Interactive Brokers really neatly. The Smart Stock Automation Engine will manage the entire trading execution process. It will:

1. Update your data
2. Run your backtests to find your signals
3. Check your account balance and position size your signals
4. Manage your exposure across multiple systems

If you want to automate your trading using Amibroker, then click this link to learn more about The Smart Stock Automation Engine. 

Best Backtesting Software

You’ve probably realized by now that I have two favorite packages for my ‘best backtesting software aware – They are Amibroker and RealTest. Most other common backtesting software is just not up to the task!

I like Amibroker because I’ve been using it for a long time. Super fast, cost-efficient, and really, really powerful. Amibroker has a great support base, there’s a lot of users out there, there’s quite a few power users and coders that can help you get the job done.

I really love RealTest because it is the only backtesting software that really does a cost effective and great job of multi-system backtesting and optimizing capital allocations between systems. TealTest also does  testing for individual instrument systems and portfolio systems. RealTest, also not terribly expensive. So it’s a great option and very easy to learn.

So, my two choices for the best backtesting software, Amibroker and RealTest. I have both, I use both. If you are starting from scratch now, either of those will do the job.

RealTest will probably take you further, but you’re probably going to find that you want both because as I say, I like to have redundancy in my trading. I’ve got multiple brokers, I’ve got my multiple trading software, I’ve got multiple computers, I’ve got multiple data providers. I want multiple everything because I want redundancy so that my trading business is as robust and stable as possible.

Today we’ll be addressing some of the most common inquiries about backtesting.

More Articles Related to Backtesting

Articles about Backtesting

• The Ultimate Guide to Backtesting
• How To Build Confidence for Your Stock Trading System
• Are you fooling yourself with faulty backtests?
• Systematic Trading Workflow
• 8 Scenarios to think 💭 beyond your trading system backtest 📈
• How Optimization Can Make or Break Your Trading System
• 1 thing you must do after backtesting your Stock Trading System
• Backtesting Your Emotions To Build Confidence In Your System
• What backtesting software should I use for my stock trading systems?
• 4 Steps to develop your own stock trading system
• Most effective way to improve your trading system’s performance
• Is End Of Day Stock Market Data Reliable?
• Learn 4 Causes of drawdown & 5 ways to reduce maximum drawdown in your trading
• Trading Systems | The Systematic Trader’s Advantage
• Should I close a trade if it doesn’t hit the profit target after X days?
• Brave New Coin – The best source of crypto data for systematic trading
• What to do when your trading system hits your maximum drawdown

Articles about backtesting with Amibroker

• How to deal with outlier trades in Amibroker Backtesting
• How To Set Up Your Amibroker Trading Software & Metastock EOD Data 📝
• The Ultimate Free Amibroker Tutorial Guide
• Amibroker Unleashed: A comprehensive step by step guide to mastering Amibroker
• Amibroker Tips: Stop The Price Data Window Disappearing

Books About Backtesting & Trading Systems

• Trading Books: ‘Trading Systems and Money Management: A Guide to Trading and Profiting in Any Market’ by Thomas Stridsman
• Trading Books: ‘Quantitative Trading Strategies: Harnessing the Power of Quantitative Techniques to Create a Winning Trading Program’ by Lars Kestner
• Trading Books: ‘Breaking The Black Box’ By Martin J. Pring

Interviews in which Adrian discusses backtesting

• Interview on The Traderwave Academy Podcast
• Interview on The Talking Trading Podcast
• Interview on The Better System Trader Podcast
• Interview on The Financial Investing Radio Podcast
• Interview on How To Trade It Podcast
• Interview on The Kiss My Money Podcast
• Adrian Interviews Peter Castle On Trading Psychology And Discusses His New Book – The Zen Trader

Other articles that mention backtesting

• What is Trend Trading and How Does It Work?
• The Ultimate Guide to Mean Reversion Trading
• The Complete Ultimate Bollinger Bands Guide
• The Ultimate Guide to Trading Stock Dividends: Maximizing Returns and Building Wealth
• The Ultimate Guide to Moving Averages
• Unlock the Power of Technical Analysis: A Comprehensive Guide
• 14 Global Stock Markets Tested for Trend Trading
• 18 Global Stock Markets Tested for Trading Mean Reversion
• Mechanical Trading Systems Save You From Yourself!
• Stock Market Trading Systems
• When Should I Switch Off A System?
• 6 Basics of Stock Trading Psychology
• Stock Dividend Aristocrats – Are They Worth your Investment
• How To Evolve As A Trader To Maximise Opportunity (& Manage Risk) In An Uncertain World
• 10 Stock Trading Myths That Could Be Holding You Back
• How to become a consistent trader?
• An Introduction to Systematic Trading
• As a stock trader, what are your thoughts on trading cryptocurrency?
• An Honest Account of a Crypto Trader’s Journey to Trading Success
• The Critical Ingredient for Long Term Crypto Trading Success
• Why even conservative stock traders should consider trading in cryptocurrency
• The Beginners Guide To Bitcoin – Everything you need to know about Bitcoin
• Are stop losses ruining your trading system profits?
• Trading Mistakes – The one thing that holds back nearly every trader on the planet
• A Model To Accelerate Your Trading Success
• What is your job when you learn stock trading?
• What Do You Love About The Stock Market And Trading?
• Learn Stock Trading | Should You Always Use A Stop Loss?
• Importance of Commission and Slippage In Trading Systems
• Should you trade with a stocks 52 week highs or lows?
• Should You Trade A Stocks 52 Week Low?
• Stock Trading Tips from Enlightened Stock Traders
• How To Be Consistent Stock Trader
• Best Trade Tracking Software That You Should Use
• Learn Stock Trading & Profit in the Stock Market With 3 Simple Secrets!
• Learn Stock Trading | Are you ready to trade stocks?
• Learn Stock Trading | The Definitive Guide To Stock Market Terms
• 12 Stock Market Statistics That Will Change Your Trade in 2021
• Real stock trading tips for traders who want control
• Bear Market Stock Trading Secrets
• The markets have lost their mind – What you should do with your stock trading…
• Writing A Trading Plan Is Not Difficult Or Time Consuming (If You Approach It The Right Way)
• 57 Trading Mistakes You Must Avoid to Become A Profitable Trader
• How To Calculate Position Sizing | What Is The Best Way?
• How to Set achievable CAGR & Drawdown targets for stock portfolio
• Can I trade multiple time frames with EOD stock market data?
• EOD Trading Signals And Your Trading System
• The Best Trading Signals In 3 Simple Steps
• 5 Steps to capture the next big stock market move with trend following
• Selecting The Right Trading Signals
• Stock Trading Systems – How to make money trading stocks
• Are you using stop losses the right way? Improve Profits Today!
• 2 Methods to Launch New Trading Systems & Manage Initial Drawdown
• What can you do about sudden overnight price gaps?
• This trading system development tip is almost too easy