Myths, corrected
13 backtesting myths, and what’s actually true.
Each of these comes up again and again in trading communities. Most of them sound sensible, and each one can cost real money.
- Myth
Myth: A long, profitable backtest proves the strategy works.
What’s trueIt can still be luck, over-tweaking, or one kind of market.
Length helps only when the test is honest. The strongest evidence is how the strategy does on data it was never tuned on.
Guide: Real edge or just luck? - Myth
Myth: Adding filters makes a strategy better.
What’s trueAdding conditions until the past looks good is exactly how curve fitting happens.
Each filter that removes a few past losers is another chance to fit noise. Fewer, simpler rules are usually more trustworthy.
Glossary: curve fitting - Myth
Myth: Tweak it until the results look good.
What’s trueEvery tweak fitted to the past makes the backtest less trustworthy, not more.
Decide what you’ll test before you see results, and stop when a change has no reason behind it other than a better backtest.
Guide: Real edge or just luck? - Myth
Myth: Testing more ideas proves more.
What’s trueThe more versions you try, the more likely one looks great by pure chance.
Get enough people to flip a coin ten times and someone will get eight heads. The bar for believing a result has to rise with every try.
Glossary: data snooping - Myth
Myth: My out-of-sample data stays out-of-sample forever.
What’s trueLook at it, change the strategy, look again, and it isn’t out-of-sample anymore.
A sealed test only gives an honest answer if it’s opened once, on the finished strategy.
Glossary: sealed test - Myth
Myth: A fixed number of trades, like 30 or 100, makes a result trustworthy.
What’s trueThose are rules of thumb, not proof.
How many trades you need depends on how big the edge is compared with the noise, how many versions you tried, and whether the trades span different market conditions.
Glossary: sample size - Myth
Myth: More years of data automatically means more confidence.
What’s trueOnly if the test itself is honest.
Bad data, look-ahead mistakes or unrealistic fills make a ten-year test just as misleading as a short one.
- Myth
Myth: A backtest replays exactly what would have happened.
What’s trueNot without realistic costs, slippage and fills.
Some tools assume no costs and perfect fills by default. Check what yours assumes before you trust the result.
Glossary: slippage - Myth
Myth: A high win rate means a strong strategy.
What’s trueA high win rate can still lose money.
What a win earns compared with what a loss costs matters just as much. Look at expectancy, not just how often it wins.
Glossary: expectancy - Myth
Myth: Live results should match the backtest.
What’s trueCosts, fills and execution always differ.
The question isn’t whether live trading matches the backtest exactly. It’s whether the edge survives the difference.
- Myth
Myth: Backtesting is useless because the past doesn’t repeat.
What’s trueIts biggest value is showing what doesn’t work, before real money finds out.
A backtest can’t promise the future, but it’s the cheapest way to rule out a strategy that never had an edge.
- Myth
Myth: Passing a prop firm challenge proves a real edge.
What’s trueA challenge tests one path through the firm’s rules.
Luck can pass a challenge, and a good strategy can fail one on a single bad day against the daily loss limit.
Glossary: prop firm challenge - Myth
Myth: AI or no-code tools mean you can trust the result without checking.
What’s trueOnly if you can see the evidence.
A tool that shows no trades, no costs and no checks gives you nothing to verify. Ask to see them.