Skip to content

Myths, corrected

13 backtesting myths, and what’s actually true.

Each of these comes up again and again in trading communities. Most of them sound sensible, and each one can cost real money.

  1. Myth

    Myth: A long, profitable backtest proves the strategy works.

    What’s true

    It can still be luck, over-tweaking, or one kind of market.

    Length helps only when the test is honest. The strongest evidence is how the strategy does on data it was never tuned on.

    Guide: Real edge or just luck?
  2. Myth

    Myth: Adding filters makes a strategy better.

    What’s true

    Adding conditions until the past looks good is exactly how curve fitting happens.

    Each filter that removes a few past losers is another chance to fit noise. Fewer, simpler rules are usually more trustworthy.

    Glossary: curve fitting
  3. Myth

    Myth: Tweak it until the results look good.

    What’s true

    Every tweak fitted to the past makes the backtest less trustworthy, not more.

    Decide what you’ll test before you see results, and stop when a change has no reason behind it other than a better backtest.

    Guide: Real edge or just luck?
  4. Myth

    Myth: Testing more ideas proves more.

    What’s true

    The more versions you try, the more likely one looks great by pure chance.

    Get enough people to flip a coin ten times and someone will get eight heads. The bar for believing a result has to rise with every try.

    Glossary: data snooping
  5. Myth

    Myth: My out-of-sample data stays out-of-sample forever.

    What’s true

    Look at it, change the strategy, look again, and it isn’t out-of-sample anymore.

    A sealed test only gives an honest answer if it’s opened once, on the finished strategy.

    Glossary: sealed test
  6. Myth

    Myth: A fixed number of trades, like 30 or 100, makes a result trustworthy.

    What’s true

    Those are rules of thumb, not proof.

    How many trades you need depends on how big the edge is compared with the noise, how many versions you tried, and whether the trades span different market conditions.

    Glossary: sample size
  7. Myth

    Myth: More years of data automatically means more confidence.

    What’s true

    Only if the test itself is honest.

    Bad data, look-ahead mistakes or unrealistic fills make a ten-year test just as misleading as a short one.

  8. Myth

    Myth: A backtest replays exactly what would have happened.

    What’s true

    Not without realistic costs, slippage and fills.

    Some tools assume no costs and perfect fills by default. Check what yours assumes before you trust the result.

    Glossary: slippage
  9. Myth

    Myth: A high win rate means a strong strategy.

    What’s true

    A high win rate can still lose money.

    What a win earns compared with what a loss costs matters just as much. Look at expectancy, not just how often it wins.

    Glossary: expectancy
  10. Myth

    Myth: Live results should match the backtest.

    What’s true

    Costs, fills and execution always differ.

    The question isn’t whether live trading matches the backtest exactly. It’s whether the edge survives the difference.

  11. Myth

    Myth: Backtesting is useless because the past doesn’t repeat.

    What’s true

    Its biggest value is showing what doesn’t work, before real money finds out.

    A backtest can’t promise the future, but it’s the cheapest way to rule out a strategy that never had an edge.

  12. Myth

    Myth: Passing a prop firm challenge proves a real edge.

    What’s true

    A challenge tests one path through the firm’s rules.

    Luck can pass a challenge, and a good strategy can fail one on a single bad day against the daily loss limit.

    Glossary: prop firm challenge
  13. Myth

    Myth: AI or no-code tools mean you can trust the result without checking.

    What’s true

    Only if you can see the evidence.

    A tool that shows no trades, no costs and no checks gives you nothing to verify. Ask to see them.