Kriptoligaschool · market · demo league
EN
Log in Sign up
BTC 75,783 −0.3%ETH 2,231 −0.7%BNB 692.69 +0.1%XRP 1.18 −5.3%SOL 89.82 −0.2%TRX 0.3184 +0.2%FIGR_HELOC 1.03 −1.0%ZEC 1,329 +6.2%HYPE 78.37 +0.2%XMR 620.34 +1.1%DOGE 0.0727 −0.0%WBT 72.76 −0.7%LEO 9.27 −0.5%LINK 10.86 +0.4%

School / Your system

Lesson 3 of 6 · 9 min

Backtesting and overfitting

Lesson goalAfter this lesson you know how to test rules on historical data, recognise overfitting and interpret a result honestly.

Backtesting means applying your rules to past data as if the market were unfolding right now. A good test does not prove that the system will work. It can, however, show you cheaply that something does not work, and that is already worth a lot.

Overfitting

Overfitting is fitting the rules to the noise in past data. The system then describes the past perfectly and predicts the future no better than chance. The more combinations you try, the greater the chance that you find something that looked brilliant only by luck.

How to test honestly

  • Split the data: you develop on one period and verify on a separate one that has not been used so far (the out-of-sample set). You use that part once.
  • Include costs: fees, slippage and, if you use futures, funding.
  • Watch for look-ahead: a decision must not use data that does not yet exist at that moment, such as the close of a candle that is still forming.
  • Watch for survivorship bias: tokens that failed are often missing from today's list, so testing only on today's winners flatters the result.
  • Include different markets: rising, falling and sideways. One big surge can hide everything.
  • Compare: is the result better than plain holding and than regular buying, at the same drawdown?

How many orders you need

A manual test without programming

You do not need programming. In training on a historical chart the dates are hidden, so you do not know what comes next. Go through it candle by candle, apply the rules and record every order in your journal as if it were real. It is slower, but it reveals gaps in the rules and teaches you to read a chart without peeking into the future.

PracticeTime MachineTrading on real history with hidden dates.

Remember

  • A backtest can show that something does not work, but it does not prove future success.
  • Overfitting happens when rules are fitted to the noise in past data.
  • Verify on an out-of-sample set, with costs and without look-ahead.
  • Few orders leave great uncertainty in the estimate of the hit rate.
  • Always compare the result with holding or regular buying.

Test your knowledge

To pass, you need to answer at least two thirds of the questions correctly. You can try as many times as you like.

1. A system gives an excellent result only with EMA47, and is negative with EMA45 and EMA50. What does this most likely mean?
2. A test uses today's 20 largest tokens on data going back five years and shows strong growth. What is the main problem?
3. On the out-of-sample set the system returns −12%, and on the development set +90%. What is the honest course of action?