Each project arrives with its purpose, milestones and first tasks written out. Here is how the first three read.
Inventory of decisions waiting on a causal answer
Most organisations carry a backlog of questions phrased as did it work: the loyalty scheme, the new onboarding call, the regional price change, the leadership course. Listing them with the decision each one feeds, who is asking and when the answer is needed shows which deserve a proper study and which only need a description of what happened.
First tasks
- List every did-it-work question you have been asked in the last six months
- Write beside each one the decision it feeds and who makes it
- Ask each owner what answer would change what they do
- Pick the three questions most worth a proper study
- Revisit the question list and drop any that have gone stale @recurring(quarterly)
Turning a causal question into a precise estimand
Asking whether the loyalty scheme increased spending hides choices that change the answer: increased it for whom, compared with what, measured over how long. Writing the estimand down, with the treated population, the comparison, the outcome, the time horizon and whether you want the average effect on everyone or on those treated, prevents a correct analysis of the wrong question.
First tasks
- Write the causal question for your first study as one plain sentence
- Name the population, treatment, comparison, outcome and time horizon separately
- Decide whether the decision needs the effect on all units or on treated units only
- Send the estimand to the decision owner and get a written yes
Record of why a randomised experiment is ruled out
Observational designs need stronger assumptions than a randomised test, so the first honest step is checking whether randomisation really is off the table. Recording the reason, whether the change already shipped to everyone, randomising would be unethical or unlawful, the unit is a whole country or the sample is too small, stops the team defaulting to a weaker design out of habit.
First tasks
- Ask the change owner whether any part of the rollout could still be randomised
- Write down the specific reason a full experiment is ruled out
- Check whether a randomised rollout order is possible instead
- Record the decision and who agreed it in the study folder