Audit of tests run in the last twelve months
Before changing how a team experiments, find out how it has been experimenting. Pulling every test from the last year out of the testing tool, tickets and slide decks, with its hypothesis, primary metric, planned sample size, duration and the decision taken, often shows that many had no written hypothesis and several were stopped the day they looked good.
First tasks
- Export the list of tests from the testing tool for the last twelve months
- Search tickets and decks for tests run outside the tool
- Mark each test for a written hypothesis, primary metric and planned sample size
- Ask the agent to summarise the most common gaps across the audit sheet