Backtesting Discipline

A written backtest protocol, point-in-time and survivorship checks, honest cost and fill assumptions, fixed test periods and a register of every variant, so a result means what it claims.

Free · Mac (Apple silicon) and Windows · v1.0.5 · Other platforms

Then open the Marketplace in the app, search for Backtesting Discipline, and click Install. It goes into your own files, and works offline from then on.

Before you start: this plan is general information to help you organise decisions about money. It is not financial, investment, tax or legal advice, and it does not recommend any product, investment or deal. Rules differ between countries, so check decisions that matter with a qualified, licensed adviser in yours. Investing and trading can lose money, and raising or managing other people's money is regulated in most countries.

What it installs.

1

Pillar

1

Area

50

Projects

200

Starter tasks

🧮 Quantitative Finance & Trading

Backtesting Discipline

Keeping backtests honest: point-in-time data, survivorship and look-ahead checks, realistic costs, fixed test periods and a written record of every variant tried.

  1. Written backtest protocol before the first run
  2. Fixed development, validation and holdout dates
  3. Variant register for every backtest run
  4. Point-in-time audit of the research data set
  5. Survivorship bias check on the test universe
  6. Look-ahead review of signal timestamps
  7. Default cost model for commissions, spreads and fees
  8. Benchmark and null portfolio beside every test
  9. Reproducible run record for each backtest
  10. Red-flag checklist for backtest results
  11. Weekly variant register review
  12. Monthly review of backtest assumptions
  13. Quarterly reference backtest re-run
  14. Peer review gate before results are shared
  15. Monthly data snapshot routine
  16. Holdout access log and one-look rule
  17. Fill assumption check against recent trades
  18. Corporate actions spot check
  19. Quarterly tidy of quoted backtest results
  20. Backtest bias catalogue with worked examples
  21. Fill timing conventions and next-bar execution
  22. Reading a backtest tear sheet critically
  23. Error bars on Sharpe ratio and drawdown
  24. Walk-forward testing practice
  25. Volume participation caps in simulated fills
  26. Dividend, split and total return adjustments
  27. Short-sale constraints and borrow costs
  28. Financing, margin and cash interest in backtest returns
  29. Parameter sensitivity sweep and plateau check
  30. Cost doubling and break-even cost test
  31. Delay-by-one-bar timing test
  32. Rebalance frequency comparison with costs on
  33. Universe rules with point-in-time liquidity filters
  34. Choosing backtesting software for personal research
  35. Written stop rule for abandoning a research idea
  36. Random entry and shuffled signal sanity test
  37. Sub-period and regime breakdown of results
  38. Backtest review meeting before a capital decision
  39. Holdout reveal for a finished strategy
  40. Annual audit of every backtest still relied on
  41. Backtest write-up for an interview or portfolio
  42. Backtest rules for a part-time personal trader
  43. Questions for a backtest someone else ran
  44. Joining a team and learning its backtest standards
  45. House backtest standard for a research team
  46. Reviewing backtests as a manager or allocator
  47. Intraday bar construction and timestamp audit
  48. Market impact assumptions for larger simulated books
  49. Planted signal test of the backtest pipeline
  50. Point-in-time fundamentals with reporting lags

Where it starts.

Each project arrives with its purpose, milestones and first tasks written out. Here is how the first three read.

Written backtest protocol before the first run

Before any code runs, write down the hypothesis, the universe, the data source, the rebalance rule, the cost model, the test periods and the single measure that decides pass or fail. A protocol fixed in advance is the main defence against quietly redefining success after seeing the equity curve, which is how most attractive backtests are made.

First tasks

  • Write the hypothesis in one sentence with the economic reason behind it
  • List universe, data source, rebalance rule and cost model on one page
  • Set the pass and fail thresholds before running anything
  • Commit the dated protocol next to the research code

Fixed development, validation and holdout dates

Choosing the test window after seeing results is look-ahead by another name. Splitting the history now into a development period, a validation period and a final holdout, each with exact start and end dates, means every later result is judged against the same yardstick instead of whichever years flatter it.

First tasks

  • Decide the development, validation and holdout date ranges
  • Check the holdout covers at least one rising and one falling market
  • Put the dates in a config file the backtest reads
  • Write the one-look rule for the holdout into the protocol

Variant register for every backtest run

The number of variants tried is the figure most backtest reports leave out, and it is the one that tells you whether the best result is skill or luck. A register with one row per run, including the failures, the quick parameter nudges and the runs done late at night, makes that number exist.

First tasks

  • Create a register with run ID, date, change made and headline result
  • Back-fill every run you can reconstruct from the past month
  • Add a rule that no result is quoted without its register row
  • Have the backtest write a register row automatically at the end of each run

Make it yours.

Install Backtesting Discipline, then tell Aurum the date you're aiming for. It checks whether the plan fits the weeks you actually have.

Free · Mac (Apple silicon) and Windows · v1.0.5 · Other platforms

Blueprints are general information to help you organise your own work. They are not medical, financial, legal or other professional advice. See the Terms.