Why are backtests so unreliable?

ppteam

If you've been looking into trading futures, commodities, forex or stocks, you've probably been exposed to "backtest" results. These results purportedly show what a trading method would have done in the past, had you followed it. They are one reason why the US government requires the phrase "past performance is no guarantee of future results" when discussing trading systems or approaches.
Backtests are hypothetical, and may or may not have been actually traded with real money. It is even possible that it would be IMPOSSIBLE to make the trades shown in a backtest.

As an experienced system developer, in 5 minutes with my Tradestation testing software, I can create a system in any market with a backtest that would blow you away - it would look that good. BUT, it would fail going forward - practically guaranteed! Trust me, many developers create systems exactly this way, and then try to sell you their "secret."

So, why are backtests so unreliable? Four reasons come to mind.

1) Optimization - Most testing software has an optimization feature, which will pick the best set of parameters, based on the past data. Most developers abuse this feature. What worked best in the past is highly unlikely to be the best in the future. Consequently, by falling for these overoptimized backtests, you think you are buying a Mercedes, but in reality you'll get a Yugo.

2) Hindsight bias - It is difficult to create a system without "peaking" at the data. Since one can't peak at future data, a developer who does this during development is, in effect, cheating. Many times, people do not even realize they are doing this - it can be that subtle of a mistake.

3) Software Limitations - The software itself has limitations that allow unrealistic or unachievable fills. For example, systems with market on close orders very likely include unrealistic fills, since the order might be sent after the market is closed (and never filled), but the software still thinks it got filled.

4) No real time performance - Developers post their own backtest results, make them appear as they are real, and have no independent real time verification of their results. Can you really trust backtests from the same person trying to sell you the system?

What's the solution to this backtest dilemma? Simple, if you are dealing with a CTA (commodity trading advisor), hedge fund or mutual fund, make sure you see actual, audited real time records. If you are dealing with a system developer, make sure the results you see are independently checked and verified, preferably with the results based on real money (not demo) accounts. I list two of these websites at the bottom.

It is easy to be seduced by extraordinary looking backtest results. Just remember that those results might not be real.

Websites such as worldcupadvisor.com or collective2.com are just two of many websites that offer third party verification of trading results.