Overfitting and Multiple Testing

A trial ledger, a sealed holdout, deflated Sharpe ratios, purged cross-validation and a written evidence bar, so the strategies you promote are less likely to be noise you selected.

Free · Mac (Apple silicon) and Windows · v1.0.5 · Other platforms

Then open the Marketplace in the app, search for Overfitting and Multiple Testing, and click Install. It goes into your own files, and works offline from then on.

Before you start: this plan is general information to help you organise decisions about money. It is not financial, investment, tax or legal advice, and it does not recommend any product, investment or deal. Rules differ between countries, so check decisions that matter with a qualified, licensed adviser in yours. Investing and trading can lose money, and raising or managing other people's money is regulated in most countries.

What it installs.

1

Pillar

1

Area

50

Projects

200

Starter tasks

🧮 Quantitative Finance & Trading

Overfitting and Multiple Testing

Controlling overfitting in strategy research: counting trials, deflated performance measures, cross-validation for time series and holding back data you never touch.

  1. Overfitting exposure audit of current research
  2. Trial ledger set-up for every backtest run
  3. Sealed holdout period and universe
  4. Research degrees of freedom inventory
  5. Pre-registration form for strategy hypotheses
  6. Written evidence bar before a result counts
  7. Choosing an error rate policy, FWER or FDR
  8. Random signal baseline for your backtester
  9. Core reading on backtest overfitting
  10. Monthly trial count reconciliation
  11. Quarterly holdout seal audit
  12. Weekly overfitting check in research review
  13. Effective number of independent trials estimate
  14. Post-promotion decay ledger
  15. Parameter sensitivity report for every candidate
  16. Weekly pre-registration habit
  17. Quarterly review of the research search space
  18. Peer red-team of a promising result
  19. Deflated Sharpe ratio worked by hand
  20. Probability of backtest overfitting with CSCV
  21. Purged and embargoed k-fold cross-validation
  22. Combinatorial purged cross-validation paths
  23. Reality check and SPA tests on a strategy family
  24. Holm and Benjamini-Hochberg on candidate signals
  25. Rank stability of variants across sub-periods
  26. Minimum backtest and track record length
  27. Walk-forward design with nested optimisation
  28. Haircut Sharpe ratios on the candidate list
  29. Cutting the parameter count of one strategy
  30. Walk-forward or CPCV as the default test
  31. Reconstructing trial counts for past research
  32. Parameter averaging instead of a single optimum
  33. Selection bias audit of universe and filters
  34. Raising the t-statistic hurdle for new factors
  35. Opening the holdout for a final candidate test
  36. Pre-promotion overfitting review meeting
  37. Research seminar with multiple-testing disclosure
  38. Year-end audit of trials and promotions
  39. Overfitting section of a model validation pack
  40. Overfitting controls for a solo researcher
  41. Academic statistics habits translated for quant work
  42. Team trial counting policy for a research pod
  43. Overfitting controls when history is short
  44. Hyperparameter search accounting for ML models
  45. Data snooping across a shared research database
  46. Synthetic market paths to test your selection process
  47. Bayesian shrinkage of backtest Sharpe ratios
  48. Expected maximum Sharpe under the null for your pipeline
  49. Estimating the false discovery rate of your pipeline
  50. Written overfitting policy for your research practice

Where it starts.

Each project arrives with its purpose, milestones and first tasks written out. Here is how the first three read.

Overfitting exposure audit of current research

Before adding any new control, find out how exposed your existing work already is. Most researchers cannot say how many variants sit behind their best backtest, which data has already been looked at, or which parameters were tuned by eye. A one-page audit of the live candidate list turns a vague worry into a ranked set of specific problems to fix first.

First tasks

  • List every strategy you currently consider promising, with its headline Sharpe
  • Estimate the number of variants run behind each one from notebooks and logs
  • Score each candidate low, medium or high for overfitting exposure
  • Pick the three highest-exposure candidates to re-test first

Trial ledger set-up for every backtest run

Every multiple-testing correction needs one number you probably do not have: how many trials were run. A ledger that records each backtest configuration automatically, with its parameters, data window and result, gives you that number for free from now on and makes later deflation honest rather than guessed.

First tasks

  • Define the ledger fields, including the returns series location for each run
  • Add a hook to the backtest runner that writes a row on every run
  • Make sure crashed and abandoned runs still leave a row
  • Write the query that counts trials and summarises results per family

Sealed holdout period and universe

A holdout only protects you if it has never been seen, and data that has been peeked at once is no longer out of sample. Choosing a final period, and ideally a slice of instruments, that no research query may touch until a strategy is ready for its final test gives you one genuinely fresh look per idea.

First tasks

  • Choose the holdout period length before checking how current ideas perform in it
  • Block the holdout in the default research data loader
  • Write the access rule for opening the holdout and who signs it off
  • Record every past occasion the holdout data was viewed or used

Make it yours.

Install Overfitting and Multiple Testing, then tell Aurum the date you're aiming for. It checks whether the plan fits the weeks you actually have.

Free · Mac (Apple silicon) and Windows · v1.0.5 · Other platforms

Blueprints are general information to help you organise your own work. They are not medical, financial, legal or other professional advice. See the Terms.