Backtesting and system design

The answer

A backtest applies a set of rules to past prices and reports what would have happened. It is useful and it is necessary.

But a backtest is not evidence that a strategy works. It is a hypothesis you have not tested yet. The data used to build a rule cannot also be the data that confirms it, and almost every retail backtest breaks exactly that rule.

Why this costs you money

You build a rule. You test it on 10 years. It returns 34% a year with a manageable worst loss. You feel a specific kind of excitement, and that excitement is the most expensive emotion in this cluster.

You put real money in. Over the next 8 months you lose 22%, which is worse than anything in the backtest. You conclude that the market has changed.

The market did not change. Here is what actually happened, and it happens to almost everybody the first time.

You did not build the rule and then test it. You tested something, saw it fail, adjusted it, tested again, adjusted again, and stopped when the result looked good. That process is called fitting, and the finished rule describes the particular 10 years you were looking at. It contains the 2008 crash, the 2020 crash and every unusual quarter in between, sanded into a shape that survived all of them.

The future does not contain those events. It contains different ones.

The second cost is the one nobody warns you about. A backtest with a bad flaw does not just fail — it fails after you have increased your position size, because 6 profitable months made you confident. The drawdown arrives at your largest size, not your smallest.

How it works

A backtest has 4 parts and each of them can quietly ruin the result.

1. The rules. Every decision must be written so a computer could execute it with no judgement. What triggers entry, at what price, how much, what closes the position, what stops it, what prevents 2 positions at once. If any step needs a person to look at the chart and decide, you do not have a system. You have a habit with a story attached.

2. The data. Prices adjusted for splits, bonus issues and dividends. Enough history. And — this is the one people miss — the list of instruments must be the list that existed at the time, not the list that exists now.

3. The costs. Brokerage, exchange fees, taxes and stamp duty. And slippage: the difference between the price you assumed and the price you would actually have received. A strategy that trades often can look excellent before costs and be a loss-maker after them. This is not a small adjustment. For a short-term system it is frequently the whole result.

4. The measurement. What you report at the end. Article 7 in this cluster is entirely about this, because reporting only the return is how people fool themselves.

Now the central idea of the whole cluster.

In-sample data is the data you looked at while building the rule. Out-of-sample data is data the rule has never seen. A result on in-sample data proves nothing, because you chose the rule knowing that data.

Think of it as an examination. If a student sees the question paper while revising, a score of 95% tells you nothing about the student. To learn anything, you have to ask a question they have not seen.

The practical method has 3 steps.

Split the data before you start. Take your history and cut it into 2 parts, usually 70% and 30%. Put the recent 30% away and do not open it. Build everything on the first 70%.

When you are finished building, test once on the held-back part. Once. If you look at the out-of-sample result and then adjust your rule, the held-back data has just become in-sample data and you have destroyed it. You cannot get it back.

Then walk it forward. Instead of 1 split, do many: build on years 1 to 5, test on year 6; build on years 2 to 6, test on year 7; and so on. Add up only the test years. This is called walk-forward analysis and it is the closest thing available to an honest simulation of how you would actually have used the rule through time.

What it tells you, and what it does not

A clean backtest tells you 1 thing: your rule was not obviously broken over this specific stretch of history. That is a screening result. It removes bad ideas. It does not confirm good ones.

It does not tell you that the future resembles the past. Every backtest silently assumes it does.

It does not tell you that you could have executed it. Backtests fill orders at prices that may not have been available in the size you assumed, on days when the stock may have been locked at a circuit limit, in companies that may have been too thinly traded to enter at all.

It does not tell you that you would have followed it. This is the largest gap and it never appears in any report. A backtest sits through a 9-month losing period with perfect composure. You will not.

And it does not tell you how many other rules you tried before this one. That number is the most important input to how much you should believe the result, and the backtest report does not contain it.

The decision rule

Treat every backtest as a hypothesis with a stated expiry.

Before you risk money, write down 3 numbers from the backtest: the worst peak-to-trough loss, the longest period without a new high, and the number of trades.

Then write 1 sentence: "I will stop this strategy if the live loss exceeds [worst backtest loss × 1.5], or if it goes [longest backtest flat period] without a new high."

Then trade it at a size where hitting that limit is survivable and boring.

The multiplier of 1.5 is a convention, not a law. The reason for it is the single most important sentence in this cluster: the worst drawdown in your backtest is a floor, not a ceiling. You saw the worst of a sample. The future gets its own sample, and it has no obligation to be gentler.

Also: paper trade first. Run the rule live, in writing, with no money, for long enough to hit at least 1 losing streak. What you are testing is not the rule. It is you.

Try this now

This is the most valuable 5 minutes in the cluster, and most people fail it, which is the lesson.

Take a rule you already believe. Something you actually say to yourself. "I buy when a stock breaks out of a range on strong volume." "I add to a position when it dips." "I get out when the trend breaks."

Now write it down so precisely that a stranger could execute it tomorrow without asking you a single question.

1. Write the setup: what makes an instrument eligible at all. Which market, which size of company, what minimum daily traded value.
2. Write the entry trigger: the exact condition, with numbers. Not "breaks out" — say what a breakout is. Above the highest close of how many days? Or the highest price? Measured at what time of day?
3. Write the exit: the exact condition that closes a winning position.
4. Write the stop: the exact condition that closes a losing one, and where it sits when you enter.
5. Write the size: how much of your capital goes into 1 position, as a rule and not as a feeling.
6. Now read it back and mark every place where a stranger would have to ask you something.

What you should see. Most people cannot complete step 2. The rule they have been trading for years turns out to contain words like "strong", "clean", "holding well" and "confirmed", none of which a stranger can act on. That discovery is worth more than any backtest.

This is not a failure of writing. It is the discovery that you did not have a rule. You had a set of impressions that felt like a rule because you have used it many times. A belief you cannot write down cannot be tested, and a belief that cannot be tested cannot be improved. It can only be defended.

Keep the sheet. Article 5 in this cluster completes it into a full system, and article 6 tells you what to do with the parameters you just invented.

Three real cases

1. Long-Term Capital Management, 1998 (United States) — the model was right about history A hedge fund staffed with extremely capable people, including 2 Nobel laureates in economics, built positions on statistical relationships that had held reliably in the historical data. In 1998 those relationships moved together in a way the historical record did not contain, and the fund lost billions of dollars in a few months. It was recapitalised in an arrangement organised by the Federal Reserve Bank of New York in September 1998. Nothing about their history was wrong. The future simply produced a combination of events the sample did not include, which is what futures do.

2. The Turtle experiment, 1983 to 1984 (United States) — a fully written rule set, handed to strangers Two commodity traders disagreed about whether trading could be taught. They recruited a group of people with no trading background, gave them a completely specified mechanical rule set, and funded them. The rules covered entry, exit, stop and position size with no discretion. Several of the group went on to successful careers. Two details matter here and they are usually left out. First, the rules were written down completely — which is exactly the exercise above. Second, reported results within the group varied widely, and the difference was in following the rules, not in knowing them. 3. The backtest overfitting literature, 2014 (United States) — how few trials it takes to produce a great backtest by accident Researchers including a group publishing in the Notices of the American Mathematical Society argued that when a strategy is selected from many trials, the reported performance is systematically inflated, and that it takes a surprisingly small number of trials to produce an impressive-looking backtest from data with no real signal at all. They proposed reporting how many configurations were tried alongside any backtest result. Almost no commercial backtest report does this, including the ones sold to retail traders.

The question that resolves it

A novice looks at a backtest and asks: how good is the result?

An expert asks: how many times did you look?

The first question has an answer printed on the report. The second determines what the first one means, and only the person who ran the test knows it. When you are the person who ran the test, you do know it — and the discipline is to write that number down before you look at the result, not after.

What would make this wrong

If backtests were reliable evidence, then strategies with excellent backtests would perform similarly when traded live. The gap between backtested and live performance is a well-documented and consistent disappointment across the fund industry. Anybody claiming otherwise should be asked for a list of every strategy they tested, not the one they published.

The honest limits are 3.

First, backtesting is not optional and this article is not an argument against it. The alternative to a flawed backtest is no test at all, which is worse. A backtest reliably identifies rules that are definitely bad, and that alone is worth the effort.

Second, some things backtest honestly. Long-horizon, low-turnover, few-parameter approaches with an economic reason behind them are far more trustworthy than short-term pattern rules, because there is less to fit and more data per decision.

Third, out-of-sample testing is not a guarantee either. If you test 200 strategies on the same held-back data, that data is now contaminated for all of them. The protection comes from restraint, not from the technique.

Carry this

• A backtest is a hypothesis, not evidence.
• Data you used to build the rule cannot also confirm it.
• The worst drawdown in the test is a floor, not a ceiling.
• Write the number of variations you tried, before you look at the result.

Knowledge check

Related

• Designing a rules-based trading system
• Optimisation, robustness and Monte Carlo
• Performance metrics: Sharpe, Sortino, expectancy and drawdown
• ← Sentiment, cycles and system testing