Online Experimentation (A-B Tests)

Experiment briefs, an evaluation criterion and guardrails, sound randomisation and sample sizes, launch and readout routines, the statistics behind honest results, and advanced designs such as holdouts and switchbacks.

Free · Mac (Apple silicon) and Windows · v1.0.5 · Other platforms

Then open the Marketplace in the app, search for Online Experimentation (A-B Tests), and click Install. It goes into your own files, and works offline from then on.

What it installs.

1

Pillar

1

Area

50

Projects

201

Starter tasks

📊 Data & Analytics

Online Experimentation (A-B Tests)

Running controlled experiments on websites and products with proper randomisation, sample sizes, guardrail metrics and honest readouts.

  1. Audit of tests run in the last twelve months
  2. One-page experiment brief and pre-registration
  3. Overall evaluation criterion for product tests
  4. Guardrail metric set for every test
  5. Choosing the randomisation unit
  6. Sample size and minimum detectable effect calculator
  7. Exposure logging and assignment records
  8. A-A test to validate the assignment pipeline
  9. Choosing an experimentation platform
  10. Central experiment registry and results archive
  11. Pre-launch checklist for every experiment
  12. Weekly experiment design review
  13. Sample ratio mismatch alerting on live tests
  14. Standard readout document for every test
  15. Prioritised experiment backlog
  16. Traffic allocation and test collision calendar
  17. Ship, iterate or stop decision log
  18. Monthly experimentation programme metrics
  19. Feature flag clean-up after concluded tests
  20. Quarterly synthesis of experiment learnings
  21. Reading p-values and confidence intervals correctly
  22. Statistical power and the winner's curse
  23. Peeking and sequential testing
  24. Variance reduction with pre-experiment data
  25. Ratio metrics and the delta method
  26. Many metrics, many segments and false discoveries
  27. Bayesian and frequentist readouts compared
  28. Novelty and primacy effects in user behaviour
  29. Testing revenue metrics with heavy tails
  30. Diagnosing a sample ratio mismatch
  31. Minimum run time and stopping rules policy
  32. Default alpha, power and sidedness settings
  33. Moving tests from client side to server side
  34. Leading indicators when the real outcome is slow
  35. Consent, privacy and ethics review for tests
  36. Interaction checks between concurrent tests
  37. Bot and internal traffic filtering for tests
  38. First end-to-end experiment from brief to readout
  39. Checkout change shipped as a staged experiment
  40. Peak season test freeze and restart
  41. Company experimentation showcase
  42. Annual experimentation programme review
  43. Landing page and email tests for marketers
  44. Product manager's first month running tests
  45. Online field experiments for academic researchers
  46. Experimenting with low traffic in a small team
  47. Re-analysing past winners nobody trusts
  48. Long-term holdout for cumulative impact
  49. Switchback and cluster tests for network effects
  50. Multi-armed bandits versus fixed-horizon tests

Where it starts.

Each project arrives with its purpose, milestones and first tasks written out. Here is how the first three read.

Audit of tests run in the last twelve months

Before changing how a team experiments, find out how it has been experimenting. Pulling every test from the last year out of the testing tool, tickets and slide decks, with its hypothesis, primary metric, planned sample size, duration and the decision taken, often shows that many had no written hypothesis and several were stopped the day they looked good.

First tasks

  • Export the list of tests from the testing tool for the last twelve months
  • Search tickets and decks for tests run outside the tool
  • Mark each test for a written hypothesis, primary metric and planned sample size
  • Ask the agent to summarise the most common gaps across the audit sheet

One-page experiment brief and pre-registration

Writing the hypothesis, primary metric, guardrails, sample size and decision rule down before launch is what separates an experiment from a fishing trip. A one-page brief, dated and frozen before the first user is assigned, stops the readout from quietly switching to whichever metric moved. It also makes design review possible, because reviewers can see the plan instead of guessing it.

First tasks

  • Draft the brief headings and share them with two people who run tests
  • Add a decision rule field for positive, flat and negative results
  • Rewrite two recent tests in the new brief format
  • Agree that no test launches without a dated, frozen brief

Overall evaluation criterion for product tests

Teams that judge each test on a different metric end up shipping changes that win on clicks and lose on revenue. An overall evaluation criterion is the one metric, or small weighted set, that every product test is finally judged on, chosen because it moves within a test's run time and predicts long-term value. Agreeing it with leadership once saves the same argument in every readout.

First tasks

  • List the metrics recent readouts used as their headline
  • Compare candidates on how quickly they move and how well they track retention
  • Write the chosen criterion's definition and source table
  • Present the choice to the product lead for sign-off
  • Revisit the evaluation criterion with leadership @recurring(yearly)

Make it yours.

Install Online Experimentation (A-B Tests), then tell Aurum the date you're aiming for. It checks whether the plan fits the weeks you actually have.

Free · Mac (Apple silicon) and Windows · v1.0.5 · Other platforms

Blueprints are general information to help you organise your own work. They are not medical, financial, legal or other professional advice. See the Terms.