Strategy testing, explained
Can you trust your backtest? Learn how to find out.
Straight answers to the questions traders ask most about testing a strategy: overfitting, luck, costs, how much evidence is enough, and why backtests fail live. Plain English, no hype.
Start here
In-depth guides to the biggest questions, with the reasoning, the warning signs and what to check.
Real edge or just luck? How to tell if your backtest is overfit
A great backtest can still be luck. Here’s how overfitting happens, the warning signs to look for, and six checks that separate a real edge from a lucky fit.
Read the guideThe questions traders ask
Gathered from trading forums, communities and search, grouped by topic. Each has a short answer and a place to go deeper.
Real edge or luck
The question behind every other question: can you trust this result?
Is my strategy a real edge, or just luck and overfitting?
You can’t tell from the backtest alone. It needs to hold up on data it was never tuned on, after you account for how many versions you tried, and with realistic costs included.
Read the guideWhen should I stop tweaking my strategy?
Decide what you’ll test before you see results, and stop when a change has no reason behind it other than a better backtest. Once you’ve opened your final test, you’re done tuning.
Guide: when to stopHow do I keep part of my data untouched for a final test?
Set a stretch of history aside before you start, build and tune on the rest, and run the finished strategy on it once. Every extra look turns it into tuning data.
Guide: keep a sealed testBacktest vs. live
Why a strategy that looked great can disappoint with real money.
Why does it work in the backtest but lose money live?
Usually one of four reasons: it was overfit, the backtest left out real costs and fills, a bug or look-ahead mistake flattered it, or the market it was built on has changed. Sometimes it’s simply a normal losing stretch that the backtest had too.
Myth: live should matchHow do I account for slippage, commissions and realistic fills?
Charge every trade the commission and fees you’d really pay, add slippage on entries and exits, and only assume fills that could have happened. Then re-run with higher costs to see if the edge survives.
Glossary: slippageCould a bug or a peek at the future be making the profits?
It happens more than people think. The classic mistake is look-ahead bias: deciding a trade with information you couldn’t have had yet, like a bar’s close. Results that look too good deserve a line-by-line check.
Glossary: look-ahead biasWill my strategy still work when the market changes?
No one can promise that. What you can do is test how it did across different conditions, calm and volatile, trending and choppy, so you know where it struggles before the market shows you.
Glossary: market regimeHow much evidence is enough
Trades, years and numbers: what actually tells you something.
How many trades do I need before I can trust a backtest?
There’s no magic number. Rules of thumb like 30 or 100 trades aren’t proof. It depends on how big the edge is compared with the noise, how many versions you tried, and whether the trades span different market conditions.
Glossary: sample sizeHow long should I forward-test before using real money?
Long enough to see the strategy behave the way the backtest said it would, across more than one kind of market. Forward testing is slow at proving an edge, so an honest backtest matters even more.
Glossary: forward testingWhich numbers matter: win rate, drawdown, profit factor, Sharpe?
No single number. Read the win rate together with the average win and loss, look at the worst drawdown and the number of trades, and treat profit factor and Sharpe as summaries of the same trades. Every one of them is easy to inflate by overfitting.
Glossary: expectancyTools, code and AI
Testing without a programming degree, and what to ask of any tool.
Can I test a strategy without learning to code?
Yes. No-code backtesters and AI tools can turn written rules into a test. Whatever you use, ask the same three things: can it show you the actual trades, the costs it assumed, and the checks it ran?
Myth: trust the toolCan AI test my strategy, and can I trust what it gives me?
AI is good at turning a description into rules and suggesting ideas. Trust the numbers only if they come from a real backtest you can inspect, not from text the AI wrote. Ask to see the trades and the checks.
Myth: trust the toolIs backtesting even worth it?
Yes, mostly for what it rules out. A backtest can’t promise a strategy will work, but it’s the cheapest way to find out that one doesn’t, before real money does.
Myth: backtesting is uselessHow do I test an ICT or discretionary strategy?
Turn it into rules first: exactly what has to be true to enter, where the stop goes and when you exit. The parts that stay “it depends” can’t be tested honestly, so make them as specific as you can.
Glossary: discretionary tradingProp firm challenges
Profit isn’t enough when the rules can end the attempt.
Is my strategy good enough for a prop firm challenge?
A challenge asks whether your strategy can reach a profit target without breaking rules like a daily loss limit or a maximum loss. A strategy that’s profitable overall can still fail on one bad day, so test it against the rules, not just for profit.
Glossary: prop firm challengeThe backtest check
Can I trust this backtest? Eight questions to ask first.
Use it on your own results, a strategy someone is selling, or anything a tool or an AI hands you.
- Does it include realistic costs?Commissions, fees, slippage and fills you could really have gotten.
- Do you know how many versions were tried?The best of many tries needs a much higher bar than a first try.
- Was some data kept aside and tested only once?The cleanest check that it wasn’t tuned to the past.
- Does it survive small changes to its settings?Real edges sit on plateaus, not spikes.
- Are there enough trades, across different market conditions?Many trades from one kind of market still tell only one story.
- Is the profit spread across many trades?If a handful of trades carry it, a handful of lucky days might too.
- Could it be using information from the future?Look-ahead mistakes make impossible results look easy.
- Can you see the actual trades?A summary you can’t check is a claim, not evidence.
Myths, corrected
Things traders repeat about backtesting, and what’s actually true.
A long, profitable backtest proves the strategy works.
What’s trueIt can still be luck, over-tweaking, or one kind of market.
MythAdding filters makes a strategy better.
What’s trueAdding conditions until the past looks good is exactly how curve fitting happens.
MythTweak it until the results look good.
What’s trueEvery tweak fitted to the past makes the backtest less trustworthy, not more.
MythTesting more ideas proves more.
What’s trueThe more versions you try, the more likely one looks great by pure chance.
MythMy out-of-sample data stays out-of-sample forever.
What’s trueLook at it, change the strategy, look again, and it isn’t out-of-sample anymore.
MythA fixed number of trades, like 30 or 100, makes a result trustworthy.
What’s trueThose are rules of thumb, not proof.
Glossary
Backtesting terms in plain English, from curve fitting to walk-forward analysis.