Each project arrives with its purpose, milestones and first tasks written out. Here is how the first three read.
Written backtest protocol before the first run
Before any code runs, write down the hypothesis, the universe, the data source, the rebalance rule, the cost model, the test periods and the single measure that decides pass or fail. A protocol fixed in advance is the main defence against quietly redefining success after seeing the equity curve, which is how most attractive backtests are made.
First tasks
- Write the hypothesis in one sentence with the economic reason behind it
- List universe, data source, rebalance rule and cost model on one page
- Set the pass and fail thresholds before running anything
- Commit the dated protocol next to the research code
Fixed development, validation and holdout dates
Choosing the test window after seeing results is look-ahead by another name. Splitting the history now into a development period, a validation period and a final holdout, each with exact start and end dates, means every later result is judged against the same yardstick instead of whichever years flatter it.
First tasks
- Decide the development, validation and holdout date ranges
- Check the holdout covers at least one rising and one falling market
- Put the dates in a config file the backtest reads
- Write the one-look rule for the holdout into the protocol
Variant register for every backtest run
The number of variants tried is the figure most backtest reports leave out, and it is the one that tells you whether the best result is skill or luck. A register with one row per run, including the failures, the quick parameter nudges and the runs done late at night, makes that number exist.
First tasks
- Create a register with run ID, date, change made and headline result
- Back-fill every run you can reconstruct from the past month
- Add a rule that no result is quoted without its register row
- Have the backtest write a register row automatically at the end of each run