The devil in Algo Trading: Overfitting !
As you enter the profession of algorithmic-trading, all you hear are the anecdotes of everyone who fell into the trap of looking at a beautiful optimised set of backtesting results and charts - that performed miraculously well - only to fail in a live environment with real capital. The catastrophic causal factor is imprinted in the brain early in one's algorithmic trading development career - overfitting.
So now as an aspiring algorithmic trader or strategy developer you're always looking over your back to gauge whether your solution is overfitted. How can you know? If the results are really good - does this mean its necessarily due to overfilling - what if your strategy is just very good. I've spent many nights in the last eight months sweating about this very topic.
From my errors, reading many papers, listening to algorithmic trading podcasts and just plain learning from failure - here is the set of rules that we have developed at Aieden Technologies to avoid overfitting:
1) Develop your strategy based on first principles
2) Independently optimise your filters - not together
3) Run live with a small amount of capital for six months+
This means that you develop your strategy and use first principles data analysis to set the parameters for said strategy. If you develop a digital filter - you use digital filtering techniques to determine the frequency of your data and where your filter frequencies need to be. This way you avoid using only optimisation as a way to determine your filter frequencies - an approach that would be prone to data overfitting.
If you have multiple filters as part of your strategy (say trend and mean-reversal filters) - ensure you develop them from first principles