Introduction
When it comes to trading system optimization, many traders unknowingly fall into common traps that lead to unreliable backtest results and poor real-time performance. In this article, we delve into the subtleties and concepts that are often overlooked, providing a comprehensive guide to optimizing your trading systems effectively.
Are Your Backtest Results Realistic?
Many traders worry about whether their backtest results are realistic. This concern arises when trading rules that worked in the past fail in real-time trading, or when systems collapse on unseen data. These problems often stem from incorrect optimization or a misunderstanding of optimization principles.
- Backtesting Concerns: Have you ever tested trading rules that seemed to work historically but failed in real-time trading? This is a common issue among traders, leading to doubts about the reliability of backtesting itself.
- Sample Size and Data Quality: Ensure your backtest includes a sufficiently large and representative sample size. Small sample sizes or data that doesn’t reflect various market conditions can lead to skewed results.
- Historical Data Limitations: Remember that backtests rely on historical data, which may not perfectly predict future market behavior. The key is to use past performance as a guide while remaining cautious about future expectations.
The Industry’s Influence on System Development
The trading industry often drives traders to failure by encouraging them to ask the wrong questions. Common questions like “What’s the best indicator?” or “What rules could I add to make my system better?” focus on past data and fail to enhance future performance. Instead, traders should ask, “What rules and parameters will give me the greatest chance of future profitability?”
Misleading Guidance
- The Best Indicator Myth: Many traders are led to believe that there is a single best indicator that can guarantee success. This mindset can lead to endless searching and tweaking without achieving consistent results.
- Parameter Tuning: Overly focusing on fine-tuning parameters based on historical data can result in curve fitting, making the system too sensitive to past conditions.
- Market Conditions: Markets are dynamic, and what worked in the past may not work in the future. It’s crucial to develop systems that can adapt to changing market conditions.
Avoiding Optimization Traps
Optimization traps are the pitfalls that can derail a trading system’s profitability. The most common trap is curve fitting, which occurs when a system is tailored too closely to past data, resulting in poor performance on future data.
Curve Fitting is the EVIL NEMESIS of system traders!
Key Causes of Curve Fitting
- Optimization Technique: Using techniques that overly fine-tune parameters to past conditions can lead to unstable profit peaks.
- Structural Effects: The interaction between system components can cause curve fitting. Each additional rule requires exponentially more data to optimize confidently.
- Trade Inclusion and Exclusion: Including lucky big wins or excluding adverse trades can mislead the backtest results.
- Market Dynamics: Designing systems based on unique market conditions that may not repeat can result in unreliable systems.
Principle 1: Asking the Right Questions
The first principle for optimizing correctly is to ask the right questions. Instead of focusing on past performance, traders should concentrate on what rules and parameters will maximize future profitability.
High-Quality Questions
- Future Profitability: Ask, “What rules and parameters give the greatest chance of future profitability?” rather than focusing solely on past performance.
- System Stability: Inquire about the stability of the system across various market conditions and timeframes.
- Robustness: Ensure the system can handle unexpected market changes without significant performance degradation.
Principle 2: Avoiding Optimization Traps
Unstable Profit Peaks
Unstable profit peaks occur when optimized parameters show high performance in backtests but fail in real trading. To avoid this, choose parameter values that are broadly representative of the surrounding values’ performance.
- Visualizing Stability: When you optimize, visualize the performance of parameter values across a range to identify stable regions. Avoid spikes that indicate sensitivity to specific conditions.
- Representative Performance: Choose parameter values that are stable and representative of the system’s overall behavior, not just the best-performing ones.
Brute Force Optimization
Brute force optimization involves testing numerous parameter combinations simultaneously, which can lead to compounded curve fitting. Instead, focus on analyzing stable performance areas and avoid relying solely on the best backtest result.
- Compounded Curve Fitting: When multiple rules are optimized simultaneously, the risk of curve fitting increases. Each rule’s optimization can stack, leading to an unrealistic system.
- One Parameter at a Time: Optimize one parameter at a time to understand its impact on the system. This approach helps in identifying stable parameter ranges.
Structural Effects
The number of trades and trading rules can significantly impact optimization. Systems with fewer trades are more prone to curve fitting, and each additional rule requires more data for reliable optimization. Simplify rules to capture general market behavior rather than specific past trades. Systems with a large number of rules and / or complex rules are highly prone to over-fitting and should be simplified.
- Trade Volume: Ensure that your system generates a sufficient number of trades for meaningful analysis. Systems with too few trades can be misleading.
- Rule Complexity: Avoid overly complex systems with many rules. Simpler systems that capture general market behavior are often more robust and reliable.
Trade Inclusion and Exclusion
Including a few big winners or excluding a few large losses can distort backtest results. Ensure that rules and filters impact a broad range of trades over a long period to avoid curve fitting.
- Big Winners: Including lucky big wins in your backtest can inflate the system’s perceived performance. Remove these outliers and re-optimize to check for stability.
- Large Losses: Similarly, excluding large losses can make a system appear more robust than it is. Ensure your system can handle adverse market conditions.
Market Dynamics
Systems designed for specific market conditions may not perform well in different environments. Test systems over diverse market conditions to ensure stability and robustness.
- Bear Markets: A system optimized for bear markets may not perform well in bull markets and vice versa. Ensure your system can adapt to various market conditions.
- Historical Events: Consider how unique historical events, like the 2008 financial crisis, impact your system. Avoid relying solely on these periods for optimization.
Principle 3: The Robustness Method
To build unshakable confidence in your trading system, focus on stability and robustness. Ensure that your system performs well across a range of parameter values and market conditions.
Stable Parameters
Choose parameters that perform consistently even when varied slightly. This indicates a stable system that can handle future market changes.
- Parameter Stability: Test your system with slight variations in parameter values. A stable system will perform well across a range of values, not just the optimized ones.
- Broad Performance Range: Ensure that the system’s performance is stable across a broad range of parameter values, indicating robustness.
Consistent Market Edge
Your system should show a consistent edge over a 20-year backtest, across all market conditions. Avoid systems that only perform well in specific periods or rely on a few large trades for profitability.
- Long-Term Consistency: A robust system should perform well over long periods, indicating its ability to handle various market conditions.
- Avoid Dependency on Outliers: Systems that rely heavily on a few large trades for profitability are less reliable. Ensure a more even distribution of profits across trades.
When we are optimizing first remove huge positive outliers so we don’t curve fit to a few large trades
Surrounding Parameter Space
Test your system across a broad range of parameter values. If most combinations are profitable, your system is likely stable and robust.
- Comprehensive Testing: Test the system with a wide range of parameter values to ensure stability. This helps in identifying the most reliable parameters.
- Relative Stability: The stability of chosen parameter values relative to surrounding values is crucial. Prioritize stable regions over absolute performance peaks.
Examples and Case Studies
- Case Study: Forex Trader “Harry”: Harry, a Forex trader, optimized his system using 360,000 parameter combinations, finding the best values for historical data. However, his system failed in real-time trading due to backward-looking optimization. By shifting his focus to future profitability, Harry’s system began to perform successfully in real-time.
- Example: Unstable Profit Peaks: A trader optimized a parameter that showed a high-profit peak in the backtest but failed in real trading. By selecting a parameter value that performed well across a range, the system’s real-time performance improved significantly.
Advanced Techniques for Robust Optimization
- Monte Carlo Optimization: This technique involves running multiple optimization simulations with different subsets of data to ensure the system’s robustness across various scenarios.
- Walk-Forward Testing: Divide your data into multiple periods, optimize on one period, and test on the next. This method helps in validating the system’s performance over time.
Practical Tips for Successful Optimization
- Start Simple: Begin with a basic system and gradually add complexity. Ensure each added rule genuinely improves performance and isn’t just fitting past data.
- Regularly Update Your System: Markets evolve, and so should your trading system. Regular updates and re-optimization can help maintain its effectiveness.
- Use Multiple Metrics: Evaluate your system using various performance metrics like profit factor, drawdown, and Sharpe ratio to get a comprehensive view of its robustness.
Conclusion
Optimizing a trading system requires asking the right questions, avoiding common traps, and ensuring robustness. By focusing on stable parameters, consistent market edges, and comprehensive testing, traders can build reliable systems that perform well in real-time trading.
To learn more about building robust trading systems, download our Trading System Confidence Cheat Sheet and watch our free training videos at bit.ly/tradingconfidence.
FAQ
During optimization should we look at all parameters as being equally weighted or should some be more importantly valued?
Every parameter that you add to the system gives you the chance to make the system more profitable in the future, hopefully, but you need to be suspicious of every one. What I really like to see is parameters that increase performance of the system significantly and also impact a significant number of trades.
Now the importance of certain parameters. Some parameters are definitely more important than others. If you add a rule or optimize a parameter, and that parameter is less stable than another, the parameter from another rule, then you want to be more cautious and skeptical of that rule of that parameter.
It’s not necessarily that one is more important than the other in a system, but we want maximum stability. And so if I add a rule and then optimize it and find that rule is unstable, then what I’ll try and do is think of another way of expressing what I’m trying to do in the system. And replace that original, that, that rule with something else and see if it’s more stable. Because I want all of my rules to be significant and stable.
It’s not that some rules or parameters are better or more important than others, so it’s not about weighting each rule differently, but every rule in your system has to carry its weight. If it doesn’t, if every rule is not significant and material and robust, then that rule should be taken out.
Should you use WalkForward Optimization?
I really like the WalkForward strategy but there’s some challenges with it. Now, depending on what style of trading you do, WalkForward is more or less usable or more or less valuable. If you trade a short term mean reversion system that generates many trades. Then using Walkforward to optimize and re optimize the system is really powerful because you can gradually adjust how the system works to fit the market as it evolves over time.
In the walk forward window, if you can get many trades so you can actually get enough data to make the decision of what parameter value to use for each optimization window then walkforward optimization is useful. But if you’re trying to walk forward with something like a long term trend following system, where your big winning trades might last 6 – 12 months or more, walk forward is really problematic.
This is because if you have even a 5 year optimization window, and a three year walk forward window, then in a five year optimization window, you don’t get that many of the big trends. You get a couple. And so what tends to happen is you end up curve fitting in the walk forward optimization, in the optimization window, you end up curve fitting And then the walk forward fails.
It doesn’t perform because you’ve curved it to the one monster trade that happened to be in that walk forward, in that walk forward optimization period. If you’ve got a five year optimization window and a three year walk forward, in that three years, you may not have enough time for those big trends to fully develop and evolve, so the out of sample ‘walk forward’ frequently looks bad.
This means you don’t get a true estimate of the performance for a long term trend following system using walk forward optimization. And what you end up having to do is needing to use really big optimization windows and really big walk forward windows. And when you do that, you actually lose the efficacy or the benefits of walk forward because you’re only getting a couple of steps in the process before you run out of data.
For short term systems, I like it, I do it. For longer term systems I tend to be a little more blunt. I want my long term systems to be very stable over a wide range of parameter values. And then, they’ve got to work in sample and out of sample on a couple, maybe two different periods of time and if they do, then I’m happy. As long as I’ve developed it carefully. In terms of adjusting my systems going forward, what I’ll do is I’ll monitor the performance of the system compared to what I would expect that system to do in the current market conditions. And if the system is obviously losing its edge, and you can look at that through backtesting your systems constantly.
Can risk per trade also be variable that can be used for trading system optimization?
Risk per trade, position size and leverage are all parameters that you can use in your system to optimize. You can change them, adjust them, but you’ve got to be cautious when you do that. And I absolutely wouldn’t do it in like a brute force optimization way.
What I would do is develop the whole system, all of the rules, optimize all the parameters and using that robustness method process that I showed you above. But I would optimize and develop all the other rules first. And do that with a very simple position sizing method, a very conservative position sizing method. Once you’ve finalized your system rules and parameter values, then move to position sizing, leverage, exposure, number of simultaneous trades.
Then optimize risk per trade, leverage and position size separately at the end. We do this because those risk and portfolio management rules are about fitting the system to your objectives. Now, you’ve got two objectives, typically: Return and drawdown. If a system has an amazing return, but has a drawdown that you can’t tolerate, it’s no good for you.
So what you’ve got to do is optimize at the end for the risk management, the position size, the exposure, the leverage, and make sure not only are the returns within your target, but the drawdowns are tolerable. And I don’t like to do that before I’ve finalized the system rules and the parameter values I’m using. You’re optimizing at that point for your objectives. Because you should know what’s your drawdown tolerance. You should know what is your catastrophic risk that you’re willing to take on. You should understand how much return is required to keep you interested in the markets and follow it.
What are the key signs of over-optimization in a trading system?
Over-optimization, or curve fitting, occurs when a system is excessively fine-tuned to historical data, leading to poor performance in live trading. Signs include a perfect equity curve on backtest data, dramatic performance drop-offs on out-of-sample data, and high sensitivity to slight parameter changes.
How can traders ensure their backtest results are realistic and applicable to live trading?
To ensure realistic backtest results, traders should use a large and diverse dataset that mirrors various market conditions. It’s crucial to perform out-of-sample testing and walk-forward analysis to validate the system’s effectiveness in unseen market scenarios. Additionally, simplicity in system design helps prevent curve fitting and enhances the system’s adaptability to changing market conditions.
What role does the choice of optimization parameters play in the robustness of a trading system?
The choice of optimization parameters is critical. Selecting parameters that should be stable and surrounded by other stable values, ensuring that slight variations do not lead to significant performance differences. This approach helps maintain consistency and reliability when the system is applied to real market conditions.
What is the importance of system simplicity in trading system optimization?
Simplicity in trading systems is crucial because it reduces the risk of curve fitting and makes the system more adaptable to different market conditions. A simple system with fewer parameters and rules is easier to test and validate, and it’s less likely to be disrupted by minor changes in market behavior. This simplicity also aids in quicker decision-making and less computational demand during live trading.
How can traders effectively use walk-forward analysis to enhance system robustness?
Walk-forward analysis is a method where optimization and out-of-sample testing are conducted sequentially over multiple periods. This technique helps confirm the system’s effectiveness over time and in various market conditions. By continuously adjusting the system parameters based on recent market data and then testing these adjustments on a subsequent unseen data set, traders can build a more adaptive and robust trading system.
Learn The 10 Commandments For Profitable Trading
Discover the 10 most important lessons from 18 years of profitable trading & reading over 150 trading books.
Get your FREE Report & 10 Training Videos NOW!
