Home / Writing / The Backtest That Lied to You
Trading with AI · · 2 min read

The Backtest That Lied to You

AI can find a perfect strategy in historical data in minutes. That is exactly the problem. A look at overfitting and how to test honestly.

A glowing price chart line that fits too perfectly, cracking at the edge of the present

Give a modern AI system enough historical price data and a few hours, and it will hand you a strategy with a beautiful equity curve. Smooth, steep, almost suspicious. It should be suspicious. A curve that perfect usually means the model did not learn how markets behave. It memorised what one particular stretch of the past looked like.

This is overfitting, and AI makes it easier than ever to commit. With thousands of possible indicators, parameters and time windows, something will always fit by pure chance. Test enough ideas and one of them will look like genius. It is the statistical version of flipping coins until you get ten heads in a row and then declaring a talent for coins.

There are habits that help. First, split your data: build on one period, validate on another, and touch a final untouched slice only once. Second, prefer simple rules with few parameters; every extra knob is another place to hide noise. Third, test across different market regimes, calm and chaotic, rising and falling, rather than one friendly era. Fourth, include realistic costs such as fees, slippage and delays, which quietly eat many paper profits.

Then comes the humbling step: paper trade before anything real. Live markets contain surprises no dataset prepared your model for.

A useful mental shift is to treat the AI as a tireless research assistant, not an oracle. Its job is to generate hypotheses and stress-test them, while your job is to be the sceptic in the room. If a result looks too good, ask what it is hiding.

This post is educational only and not financial advice. Markets carry risk, and past performance, however pretty, promises nothing.

Share