Methodology

Carrie

Source: http://www.qi-research.com/cgi-b ... id=methodology&
Methodology
Overview
- Testability and Falsifiability
- What makes a good system?
     - Profitability
     - Temporal stability
     - Invertibility Survivorship bias
     - Simplicity
- What is a good time frame?
- Recommended books

Overview:
We use science in almost every area of our lives, but rarely we apply scientific methods to the most important things such as financial decisions. In investment and trading, most of us still rely on gut feelings and emotions. Even a lot of professional traders attribute their success to their instincts and failures to bad luck. Vast majority of readers would probably agree that trading decisions are more of an art than science. While I do not deny that there is value in good judgment and common sense, but far too often there is way too much emphasis has been put in art part, and not enough in science. This is partly because most successful traders do not want you to think their trading decisions can be emulated by a machine algorithm. I believe it can, although not easy. In more practical terms, I think the machine can emulate parts of it even if not all. Most of successful traders follow a simple set of principles and use their unbiased judgment or gut feelings to make the final decision. It is never clear if it those rules or principles they follow that does them any good or their gut feelings or combination of both. But one thing is clear, the gut feelings or instincts are not testable, only rules that spell out exactly can be tested for their validity. Everything else can simply be the result of randomness or luck. And the one who bets his money on luck is called a gambler.
However, I do believe human intuition is valuable for making trading decisions, but with the fact that the market is always uncertain, and the future is always unknown, a proven trading system will bring the reliability and peace of mind to you and avoid making irrational mistakes. Human intuition is also useful in devising such a trading system by recognizing patterns price charts or technical indicators. Testing such a trading system is much of an art too. To many it is a boring and thankless task where it does not bring any income nor the excitement of real money trading, and that is why system testing is the least developed area of trading. Too many trading systems are now in use only after being tested in no more than 5 year of historical data and only on current existing stocks (survivorship bias - a unique concept I will discuss below, which is pending patent), in a single market or commodity, and one sided trading (either only long or short). A lot of system can be found to work in a limited time period with such a restricted range of variables, but they will fail miserably in real trading when any one of the variables change, such as market stays flat instead of going up or down, or a stock being de-listed. I also recognize that there are a lot of things in investing that are not testable such as news and much of the fundamental analysis simply because there is no or lack of historical data. But I do the best what can be done. If trading system based sole on historical price and volume data can pass the most rigorous and stringent testing, combining with other factors such as fundamental analysis and good judgment will generate even better returns.

Testability and falsifiability
The wikipedia defination of falsifiability is this. A lot of people claim their systems work but it depends on their intuition. This is not a provable system. If it does not work, it must be that they are having a bad day, or their emotions clouded their judgment so they have to do more meditation. If it works, they claim it is the combination of the system design and their right "gut feeling". While I do not discount the importance of good judgment and common sense, putting too much reliance on one's intuition in trading defeats the purpose of having a system in the first place. I believe a good system should be able to be profitable albeit small profit, by itself without any human intervention. Even if you leave it running as an automated machine, it should be able to profit albeit small, after many many trades over a long period of time. A good trader intuition on top of this good system will then truly give you a splendid return. But first you must be able to prove the system is good. That is why a system must be tested by itself first completely automated fashion without any human judgment. In order to do that, the system must be specified in complete detail, a phrase like "if I see market trending up, I wait for the price bars approaching trend line and wait for opportunity to buy" is impossible to work with. First you have to define what you mean by market is trending up, is it staying above 50, 100 moving average, or William's % R, or what exactly? "Then price bars approaching the trend line" contains probably 2000 lines of code. First what is a trend line? define an algorithm or formula for identifying it, then define "approaching". You see what I mean, it is not a simple task to translate human intuition into a computer program, even for an experience programmer, it takes considerable amount of work to do so. That is why I do not use any of the retail software for system verification because they are just not flexible enough to write my own ideas in there.

What makes a good system?
Having established the fact the system must be completely mechanical in order to be testable, this section discusses the goals of a good system and thus the exact kind of tests that should be applied to measure it.

1. Profitability:
This is an obvious one. One tends to over-simplify it by looking at absolute returns, but it is a little more complicated than that.

1.1 Rate of % change:
This is the first thing I look at. This is different from the average % change, which is the sum of all percentage gains and losses of all individual trades divided by the total number of trades, or the expected percentage return. Rate of % change is expected percentage return divided by average duration or number of days. It is like the rate of growth of your portfolio. In other words, it is expected % return per trading day. It is important because it gives you so much information in one number. It literally tells you how fast you can grow your capital base. Often times one can optimize a system to generate a huge % return, but at an expense of extending the time frame, so the rate of growth is actually same or even less. If this number is too small or negative, there is no need for proceed any further.

1.2 Profit factor:
This is simply the total gain divided by total loss. If this factor is more than 1, the system is profitable, less than 1, you will have loss. The larger the number the better. It gives you an idea how stable or volatile the system is. If you have a close pf close to 1, and large average percentage change, this means you have a lot of big gains but also a lot of big losses. This is not good because there is a good chance you have a positive average percentage change by random luck because a few good trades can tip the balance. Then you have to check the next set of results.

1.3 Average % gain / average % loss:
This shows how volatile the system is. Often you will find trying to make higher % gain you will also increase average % loss. Even if the overall difference gets more positive you will still have to pay attention to large % loss to see if you can really tolerate this kind of volatility, and adjust your trading strategy accordingly, such as dividing your trading capitals into many positions and temporally too. In highly volatile system, you will have to have enough capital to withstand a number of losing streaks on a unlucky scenario.

1.4 Sample size:
If your system only generated 100 trades on S&P 500 symbols over the last 15 years, it is not a good system. For two reasons. First one is the law of large numbers in statistics simply means the testing results is inconclusive with such a small number. They could simply been lucky trades. Reason two is that you will have too few trading opportunities. This depends on your time frame, but I beleive reasonable time frame should not be longer than 40 trading days on average. (see time frame discussion). A good rule of thumb is that you should have at least 1000 trades in your sample to have any degree of confidence in your results. I typically use 1000 - 2000 sample size to verify minor changes, or parameter sweep simply only of it saves time. But I would use 10000+ samples to do the final verification or making any major changes to the system.

1.5 win/loss ratio:
This is not so important in absolute terms, but more of a personal preference. You can have a system that has 90% winner, and 10% losers, but for every winner, you gain 2%, and losers you lose 50%, you will still have a losing system. Or you can have a system with 10% winners, and 90% losers, but for every winner, you gain 50%, and losers you lose 2%, you have a winning system. It simply is matter of choice if you want to win more often but small, or win less often but big when you do. What matter is the final expected value of average percentage change already discussed in 1.1. With one exception is with application of options strategy, higher win/loss ratio will work in your advantage because options limit the losses to its premium. But for it to work your average duration or time frame must be small enough.

1.6 Average duration:
This is the average number of trading days that you are in position. Not always shorter the better because it will cost more commissions to do more frequent transactions, but too long will reduce the reliability of your testing too. This is somewhat related to time frame you choose. But in general, you do not want to want this to be longer than 20 - 50 days max. You will run into danger of turning into a long term investor (see time frame discussion). It will also reduce the sample size. It generally takes much longer time in past data to make any prediction into a shorter time in the future. For example, if you are using 100 day moving averages, then you will need look at past 100 days. Furthermore, when designing and testing a trading system, in order to achieve a reasonable level confidence, the laws of large numbers requires even longer price history so you can prove the system works. So if you are testing a 25-day lookback to predict next 5 day system, and requires 500 samples, you will need at least 500 x 30 = 15000 days (or 58 years) of price history at least assuming every price point fits your pattern, which in unlikely, in practice you probably get pattern fit 1/5 of the time, so you need 300 years of price data? It is clearly not possible for all symbols, so we run the test in parallel for many stocks, with shorter history. So if you have 100 stock with 10 year history, you get 1000 year worth of data, enough to verify your 5 day forecast system. Now you see what amount of data and work you need to design and test a system that can forecast 1 year into the future? That is why I do not believe long term forecast is viable and I do not chant the ever-so-familiar prophetic phrase "I believe the market will go up in long term, in 10 to 20 years in the future, because the market has been up for last 50 years", as many mutual fund salesmen say. I can only predict price movement with a reasonable scientific certainty in the next 5 to 20 days. But I always keep an open mind if someone comes along with a scientific method for long term forecast, I am more than happy to learn about it.

2. Temporal stability:
A system must be repeatable in any time frame, and at any point in time: This is to make sure system does not "fit the curve". "Curve fitting" is a kind of system that works well in backtesting during a specific period in time, because it is "rigged" to fit the data from the past, but the performance decreases significantly if the market condition changes. A temporally stable system should work in any market trend, and the best way to verify it is to apply the system in many snapshots in the history and must produce consistent results. One way to optimize a system without running a risk of curve fitting is to limit the number of parameter variables (usually less than 5), and make sure the system is simple and not overly complicated. Another way is to perform double blind test. This is using part of historical data to optimize system's parameter variables, but then discard results from that part of history, and run the test on the historical data the system has not seen before. For example, if you have 15 years of historical data. Use year 1 to 6 to optimize the system, then test it from year 7 to 15. If the system depends on long term market trend, then a trend detection must be built in and self-adjust according to market trend.

2.1. Diversification:
The most effective way to make sure the system is stable during market correction or directional changes is to diversify. In real life trading, one should diversify into 10 - 20 positions. Ideally, these positions should be as un-correlated as possible, such as different industry groups and sectors. But MOST IMPORTANTLY of all is directionally balanced. which means there should be roughly equal number of shorts and longs. Again this is to make sure if the whole market suddenly changes direction because of a major political or natural event, it does not bankrupt you in a very short time. A good INVERTIBLE system should be able to profit in bear or bull market. Invertibility is explained below.

3. Invertibility (*Important*):
A system that performs well long must also be tested shorting the same market during the same time period, or vise versa. And then the average of both long and short performances is the true measurement of how well the system performs. This is probably the most strict criteria unique only to my testing methodology where almost other systems that perform well when tested by others would fail at my testing. I have read numerous books in trading system design, and authors cover all potential mistakes and biases, but they all missed this one: what I call (survivorship bias. I believe this is the deadliest bias that lead many traders to incorrect assumptions about their test data. That assumption is that long works better in long run the short, that most stocks or disproportionately large percentage of stocks go up by a lot in long run than going down.
Here is what I mean by survivalship bias: Survivalship bias: Test data usually consists of stocks still exist today, and does not include any of the bankrupt companies that got de-listed. This leads system designer to optimize their system to be more bullish than it should. For example, system designer would set deeper stoploss for long positions hoping for recovery and less stop loss for shorts fearing for further bullish moves.
I have seen again and again people claiming simple momentum based trading works, but problem is they only test one side of the story: long. But after taking a closer look, the winning results is not consistent throughout time. The system generates profit in bull market and losses in bear market, and since over the last 50 years or so we had more bull market than bear market, the overall performance is positive. So this is their hidden assumption. If I reverse their system and test short trades only, the losses exceeds the gains made in long side. To me they are no different than buy and hold guys. Here is why: The only way for their method to work depends on the fact that stock market has been up for last 50 years or so. If we were in a flat market for last 50 years or down for the last 25 years, their method would have failed in any time frame. Even if you believe the market will continue its ascend over the long term, can you really stand losing streak for 2 years or 5 years if we are in a recession? The whole point of doing short term trading is to capitalize on short term moves and be affected as little as possible by the long term trend. But testing shorting on a bull market is equivalent as long in a bear market.
I first make sure the system works on short side, then invert it into long side without changing anything else, and it should work equally or better. A true measure of performance is to take the average of short trades and long trades. For example, if a system produces fewer number of short trades than long trades, and average of long trades is better than average of short trades, then we must not take average of all of them because it is biased to long side. we take average of long trades first, then take average of short trades, and then add them divided by 2. This makes sure there is no bias towards either long or short trades.
An invertible system also has advantage that it removes the assumption that market will tend higher or lower in long run, and dependency on long term market direction. If a system pass this test, it can be used blindly without care of overall market direction as long as you have equal number of shorts and longs at the same time. There is no way to know if overall market is going to keep going up at pace we have seen in the last 50 years so making such an assumption is not a reliable trading practice.

4. Simplicity:
This is an immeasurable quality. A system must not be too complex. This is the objective part. Part of system should not be purely observational, and should be based on reasons why it works not just because it worked before. Again this is to avoid the curve fitting problem. "Curve fitted" system has many tweekable variables so that designer would sweep these variable parameters until they fit exactly on the test data so it optimizes the performance. But it will not work in real trading, or even if you just change the test data to a different market the results will become drastically different. This is a typical symptom of overly complicated system. because it is "rigged" to fit the data from the past, but the performance decreases significantly if the market condition changes. As a rule of thumb, a system should not be have more than 5 variables to optimize.

Time frame
In theory, there is no one best timeframe in which to trade. You can look at any chart with time axis taken out you will not know what time frame it is in. It could be a 5 minute chart to 20 year chart. It is like a fractal pattern, the shape of the charts are the same if you keep zooming in. You have your choice. You can trade on an end-of-day basis, like most people do using daily charts. You can even trade weekly charts for longer term position trading. You can go the other way and day trade - 60 minute, 30 minute, all the way down to 5 minute charts. The key is to pick a timeframe that is suitable for your own personal trading goals.
One end of the spectrum is long term investing - usually holding a stock for more than a year, sometimes 5 years, 10 years or even a life time. This time frame violates the laws of large number rule I discussed above. There are so many things can go wrong in such a long time period and if one affects the stock you are holding you will suffer a substantial loss. You are basically risking several years of your life and money in one trade. I hear many investors who suffered major losses during the recent market downturn say "buy and hold" means "invest and die"! as they regrettably took in advise of mutual fund or retirement fund advisor that investing long term is the wise thing to do. To me long term investing is synonym to a religious belief, where you cannot disprove it. To prove or disprove any long term prediction it takes a long time!
It generally takes much longer time in past data to make any prediction into a shorter time in the future, usually many times longer. For example, you should not be using 5 day trendline to predict price moves in the next 5 days, you need look at past 25 days at minimum when trying to forecast next 5 days. So if your time frame is next 30 years, you better have 150 years of data of historical data. Even so if you did have a convincing historical data that market has been rising for the last 150 years, and you bet it will be for the next 30 years. It is a one trade you are betting everything into. I just do not see the logic in this kind of investing that so many financial advisors are advocating.
The other end of spectrum is day trading. If done correctly this is the most reliable way of trading and risk is at the minimum. But it is a full time job and defeats the purpose of trading which is to maximize the profit with minimum effort. It has been well-documented that if a trader has not been successful trading daily charts, it is highly unlikely that that same trader will be successful day trading. The fast pace of day trading, requiring instantaneous analysis and decision making in a market environment offering limited moves within the timeframe being traded is so demanding that very few traders can hold emotions at bay and trade in a disciplined fashion.
Swing trade is somewhere in the middle, typical duration of a trade is from days to weeks, usually no longer than 3 months. This time frame generates large enough sample size for system verification and but does not require the constant attention as day trading, and gives you freedom to do other things in your life besides trading. This takes much burden off your emotion and enough time to clear your head between making trade decisions. You also do not have to worry about commissions and slippage as much as day traders.

Recommended books
Marcel Link, High probability trading
Thomas N. Bulkowski, Encyclopedia of Chart Patterns
Robert Pardo, Design, Testing, and Optimization of Trading Systems
Stan Weinstein, Stan Weinstein's Secrets For Profiting in Bull and Bear Markets
David R Aronson, Evidence-Based Technical Analysis: Applying the Scientific Method and Statistical Inference to Trading Signals

snowball2009

Thanks.

Lionking