Optimisation, robustness and Monte Carlo
The answer
Optimisation means searching for the parameter values that made a strategy perform best in the past. It always finds something, including when there is nothing there.
The result of an optimisation is the highest point of a search, not a measurement. To find out whether the strategy is real, you have to test the chosen version on data the search never touched, and then simulate the outcomes it could have produced but happened not to.
Why this costs you money
Here is the sequence, and it is almost universal.
You write a moving average crossover system. The 20 and 50 combination returns 9% a year. Disappointing.
So you test 10 and 40. Then 15 and 60. Then 25 and 55. You build a grid: every short average from 5 to 50, every long average from 20 to 200. That is roughly 4,000 combinations. Your software runs it in 2 minutes and hands you the winner: 23 and 111, returning 28% a year.
You feel that you have found something. You have not. You have found the maximum of 4,000 random-ish numbers, and the maximum of 4,000 numbers is large whether or not any signal exists.
You trade 23 and 111 with real money. It returns 4%.
The specific loss is not the poor return. It is that you now believe the market changed, so you re-optimise on recent data, get a new pair, and repeat. This loop can consume 5 years. People in it feel busy and productive the entire time, because there is always a test running.
The second cost is the more dangerous one. A curve-fitted system does not fail gently. It fails at the moment conditions differ from the fitted period, which tends to be a moment when everything else is also going wrong.
How it works
Why optimisation seduces. Every parameter you tune adds a dimension to the search. With 1 parameter and 20 values you take 20 samples of history. With 3 parameters you take 8,000. Somewhere in 8,000 samples of any noisy series there is a beautiful equity curve, and your software will find it and present it without any warning that it is the best of 8,000.
This is the same problem as the seasonality problem in article 2 of this cluster, wearing engineering clothes.
Curve fitting, defined plainly. A curve-fitted system is one whose rules describe the specific accidents of the test data rather than any repeating behaviour. The clue is not in the returns. It is in what happens when you move the parameters slightly.
Parameter sensitivity is the best single test you can run, and it is free.
After optimising, do not look at the winner. Look at its neighbours. Plot the performance of every parameter combination you tested.
- If the good results sit in a broad plateau — 23 and 111 returns 28%, and 21 and 105 returns 26%, and 26 and 118 returns 27% — the strategy is describing something real, because it does not depend on the exact numbers.
- If the winner is a lonely spike — 23 and 111 returns 28%, and 22 and 110 returns 4%, and 24 and 112 returns 2% — you have found an accident. No real market behaviour turns on the difference between 110 and 111 days.
Always take a value from the middle of a plateau, never the peak. The peak is where the luck was. The middle of the plateau is where the mechanism is.
Walk-forward analysis. The proper way to optimise, and the only one that simulates what you would actually have done.
Divide history into blocks. Optimise on block 1, then trade those parameters, untouched, through block 2 and record the result. Then optimise on blocks 1 and 2, trade through block 3, record. Continue to the end. Add up only the traded blocks.
This produces a much lower number than a single full-period optimisation. That lower number is the honest one. It includes the cost of choosing parameters using only information you would actually have had, which is the cost the naive backtest hides. Cross-validation is the same idea from statistics: hold out different slices of the data in turn, fit on the rest, test on the held-out slice, and average the held-out results.
Monte Carlo simulation. Now the second half, and this is the part that changes how people size positions.
Your backtest produced a specific sequence of trades. That sequence is 1 of many sequences those same trades could have arrived in. The order was luck.
A Monte Carlo simulation takes your list of trade results and shuffles it, many times, computing the equity curve and the worst drawdown for each shuffle. What comes out is not a number but a distribution: here is the range of worst drawdowns this strategy could plausibly have produced.
The finding is consistent and it is the reason for this article. The drawdown in your actual backtest usually sits in the better half of the distribution. The median simulated worst drawdown is typically larger than the historical one, and the worst 5% of simulations are far larger.
Which gives the sentence to carry out of this cluster:
Your worst historical drawdown is a floor, not a ceiling.
It happened to be that size because the losses happened to arrive in that order. Reorder the same trades and you get a worse one. Then add the fact that the future will contain trades your sample does not, and the honest planning number is well above anything you have seen.
Risk of ruin comes from the same simulation. Run 10,000 shuffles at your intended position size and count how many hit a loss level that would end you — either literally, or the level at which you would stop. If that count is above a few percent, your size is wrong. Not your strategy. Your size. Robustness across markets and periods. A final test, and a demanding one. Run the same parameters, unchanged, on a different instrument, a different market and a different timeframe. A rule capturing something real about how prices behave usually works somewhat in several places. A rule that only works on 1 instrument in 1 period, with 1 exact parameter set, is a description of that instrument in that period.
What it tells you, and what it does not
Walk-forward and Monte Carlo tell you what a strategy might do if the future resembles the past in character, meaning similar volatility, similar liquidity and similar market structure. That is a large assumption and everything here rests on it.
Monte Carlo tells you about sequence risk. It does not tell you about events outside your sample. Shuffling 200 trades from a calm decade will never produce a crash your data does not contain.
Standard trade shuffling also assumes each trade is independent of the last, and for many strategies that is false. Trend-following losses cluster: they arrive together, in the same choppy market. Shuffling breaks the cluster and therefore understates the real drawdown. There are methods that preserve clustering, such as resampling blocks of consecutive trades rather than single trades. If your strategy has streaks, use one.
And no amount of simulation fixes a strategy that was selected from a large search. Monte Carlo on a curve-fitted system gives you a precise distribution of fictional outcomes. The order of operations matters: honesty about the search first, then simulation.
The decision rule
Never trade the peak of an optimisation. Trade the middle of a plateau, and only if the plateau exists.
Then apply 4 tests in this order. Each one is a stop, not a score.
1. Plateau. Do the neighbouring parameters also work? If no, discard.
2. Out-of-sample. Does the chosen version work on data the search never saw? Test once. If no, discard — do not adjust and retest.
3. Monte Carlo. Shuffle the trades. Take the drawdown at the worst 5% of simulations, not the historical one, as your planning number.
4. Size. Choose a position size at which the number from step 3 is survivable and does not make you stop.
And write down, before any of this, how many configurations you tested. That number belongs on the same page as the result, permanently.
Try this now
Two parts. The first takes 2 minutes and almost nobody has ever done it. The second takes 3 minutes and needs a spreadsheet.
Part 1: count your search.
- Write down every trading rule, indicator setting, strategy or approach you have tried in the last 2 years. Include the ones you tried for a week and abandoned. Include the indicator settings you changed. Include the strategies you tested but never traded.
- Count them. Write the number at the top of the page.
What you should see. Most active traders reach somewhere between 15 and 60. Almost nobody has ever counted, and it is the most important number in their research. If you have tried 40 approaches, then finding 1 that worked well over a stretch is exactly what you would expect from pure chance. Your current strategy is not a discovery. It is the survivor of a search you never recorded. Article 8 in this cluster gives you the arithmetic for how many false winners a search of that size produces.
Part 2: shuffle your own trades.
- Export your trade history from your broker's reports section. Put the profit or loss of each closed trade into 1 column of a spreadsheet, in date order. You need at least 20 trades; 50 is better.
- In the next column, build a running total — the cumulative profit and loss after each trade.
- In a third column, compute the drawdown: the running total, minus the highest running total reached so far. Take the most negative value. That is your actual worst peak-to-trough loss, in money.
- Now add a column of random numbers with
=RAND(), select your trade results with it, and sort by the random column. This shuffles the trades into a different order. - Recompute the running total and the worst drawdown. Write it down.
- Press the recalculate key to generate new random numbers and sort again. Repeat 5 times, writing down each worst drawdown.
What you should see. Your 5 shuffled drawdowns will vary a lot, and several will be worse than the one that actually happened to you. The same trades, the same win rate, the same expectancy — only the order changed.
That is the finding, and it is worth sitting with. Your actual worst loss was not a measurement of your risk. It was 1 draw from a distribution, and you have now seen 5 more draws from the same distribution using your own money and your own decisions.
Now the question that matters. Look at the worst of your 6 numbers. If that had happened, in that order, would you have continued trading? If the honest answer is no, your position size is too large today. Not tomorrow. Today.
Three real cases
1. Technical trading rules tested as a universe, 1999 (United States) — the correction for how many rules were searched Researchers tested a very large universe of technical trading rules — filter rules, moving averages, support and resistance breaks, channel breakouts — on long-run American index data, and applied a statistical method designed to account for the fact that the best rule was selected from thousands. The best individual rule looked impressive in isolation. After adjusting for the size of the search, the evidence for it was far weaker. This is the single most relevant published study to what you do in a backtesting package on a Sunday afternoon.
2. The backtest overfitting result, 2014 — how few trials it takes A group of researchers publishing in a mathematics journal argued that backtest performance is systematically inflated when a strategy is selected from multiple trials, and that a strategy with no real edge can be made to show a strong apparent result with a surprisingly small number of configurations tested. They proposed reporting the number of trials alongside every backtest, and developed an adjusted performance measure that penalises the size of the search. Commercial backtesting software sold to retail traders does not, as a rule, report this.
3. Long-Term Capital Management, 1998 (United States) — robustness across regimes The fund's positions were robust across the historical periods in its data. They were not robust across the period that arrived. The relationships it relied on had held through many market conditions and then moved together during a global liquidity event. Every test in this article except 1 would have passed. The exception is the last one: would you survive if the relationships you rely on all failed at once? That is a question about size, not about statistics, and it is the only protection that works against an event outside your sample.
The question that resolves it
A novice asks: which parameters performed best?
An expert asks: how wide is the region where it works at all?
The first question always has an answer, even on random data. The second is the only one that distinguishes a mechanism from an accident, and it is answered by looking at the results you did not choose.
What would make this wrong
If optimisation were harmless, then optimised strategies would perform in live trading roughly as they did in the tests that selected them. The consistent finding, across both academic work and the experience of the fund industry, is that they do not.
The honest limits are 3.
First, some optimisation is necessary and unavoidable. You have to choose a lookback period. Refusing to optimise means choosing parameters arbitrarily, which is not more scientific — it is the same choice made with less information. The distinction is between choosing a plateau and choosing a peak.
Second, Monte Carlo can create false comfort. A carefully computed distribution of drawdowns looks authoritative and it is built entirely from your sample. If your sample is a calm 10 years, your simulated worst case is a calm worst case, computed to 2 decimal places.
Third, walk-forward has its own failure. If you run 30 walk-forward configurations and report the best, you have simply moved the search up 1 level. The technique protects you only if you use it once.
In India
Three Indian conditions change the practical work.
Shorter, thinner history raises the risk of fitting. Reliable retail-accessible Indian equity data covers a much shorter span than American data, and clean point-in-time data on index membership and delisted companies is difficult to obtain. Fewer observations means a smaller search produces a false winner. The correct response is not to give up. It is to use fewer parameters than an American source would recommend, and to weight index-level and derivatives data, which is complete and public, over single-stock data, which is not.
Structural change is recent and large. Indian market microstructure has changed substantially over the past 2 decades: the growth of the derivatives market, changes to settlement cycles, changes to expiry schedules, changes in transaction taxes and the arrival of very large domestic institutional flows. A strategy optimised across a period containing 1 of these breaks has been fitted partly to a market that no longer exists. When you split data for walk-forward testing, try to split at the structural breaks rather than at even intervals.
Costs dominate short-term optimisation. Securities transaction tax, exchange charges, stamp duty and GST make Indian short-term trading expensive relative to several other markets. An optimisation run without full costs will systematically choose higher-frequency parameter values, because frequent trading looks free. Put costs into the optimiser itself, not into a review afterwards, or the optimiser will select against you.
In the United States
American conditions make this work easier and the temptation stronger.
Long, clean, cheap data. Free daily data covering many decades, and paid databases that include delisted companies and point-in-time index membership. This removes the biases that plague Indian testing and replaces them with a different problem: with 90 years of data and fast software, an individual can run a search of a size that would have required an institution 20 years ago.
Powerful free tools. Open-source backtesting libraries, free data interfaces and cloud computing mean a retail participant can test millions of configurations in a weekend. The number of tests you can run has grown far faster than the amount of data available to test on, and that ratio is the whole problem in 1 sentence.
A large published literature to borrow from. American academic work provides strategies with stated economic reasoning, which is a better starting point than a parameter search. It also has its own multiple-testing problem, documented by researchers who counted several hundred published return-predicting factors and argued that the standard significance threshold was far too weak given how many had been tried.
Where they differ, and what that tells you
The 2 markets fail in opposite directions, and knowing which one you are in tells you which mistake to guard against.
India's constraint is data. America's constraint is discipline.
An Indian researcher runs out of trustworthy history before they run out of ideas. An American researcher runs out of ideas long after they have exhausted what their data can honestly support.
What that tells you is where to spend your effort. In India, the highest-value work is verifying that your data is clean: adjusted correctly, including delisted companies, and free of survivorship bias. Getting that right is worth more than any optimisation. In the United States, the data is already clean, so the highest-value work is restraint: deciding in advance how many configurations you will test, and writing that number down before you start.
There is 1 conclusion that holds in both places. The number of things you tested is a required input to interpreting the thing you found. Neither market gives you it automatically, and neither market's software reports it. You have to count.
Carry this
- The result of a search is the maximum of many samples, not a measurement.
- Take the middle of a plateau. Never the peak.
- Your worst historical drawdown is a floor, not a ceiling. Shuffle to see why.
- Count how many configurations you tested and write it beside the result.