Predictive Modelling

Framing the decision, leak-free training data, honest splits and baselines, calibrated and explained models, validation reports and sign-off, and the routines that keep predictions trustworthy once people rely on them.

Free · Mac (Apple silicon) and Windows · v1.0.5 · Other platforms

Then open the Marketplace in the app, search for Predictive Modelling, and click Install. It goes into your own files, and works offline from then on.

What it installs.

1

Pillar

1

Area

50

Projects

198

Starter tasks

📊 Data & Analytics

Predictive Modelling

Building models that predict outcomes such as conversion, risk or demand from historical data, and validating them before anyone relies on them.

  1. Prediction problem framing document
  2. Target variable and outcome window definition
  3. Point-in-time training dataset snapshot
  4. Label leakage audit before the first model
  5. Train, validation and test split plan
  6. Naive baselines the model has to beat
  7. Evaluation metric chosen for the decision
  8. Cost matrix for false positives and negatives
  9. Exploratory review of the training data
  10. Interpretable regression benchmark model
  11. Modelling experiment log
  12. Model inventory with owners and status
  13. Monthly scored versus realised outcomes review
  14. Quarterly model revalidation pack
  15. Annual keep, rebuild or retire review
  16. Peer review of modelling work
  17. Label quality spot checks
  18. Override and feedback log from score users
  19. Model risk register
  20. Gradient boosted trees in practice
  21. Probability calibration skills
  22. Cross-validation designs for real data
  23. Hyperparameter tuning without fooling yourself
  24. Model explanations with SHAP and partial dependence
  25. Imbalanced outcome techniques
  26. Predictive modelling reading programme
  27. Weekend practice on public prediction challenges
  28. Survival models for time-to-event outcomes
  29. Decision threshold set from the cost matrix
  30. Input pruning for a leaner model
  31. Champion versus challenger offline comparison
  32. Interpretable or complex model decision record
  33. Evaluating a vendor's predictive score
  34. AutoML trial against the hand-built model
  35. Prediction intervals for regression outputs
  36. Model validation report and sign-off
  37. Model review board presentation
  38. Shadow scoring pilot before go-live
  39. Go-live handover pack for engineering
  40. Post-mortem of a model that underperformed
  41. First predictive model for a spreadsheet analyst
  42. Inheriting an undocumented model
  43. Small-sample clinical or academic prediction model
  44. Software engineer moving into predictive modelling
  45. Client predictive model as a consultant
  46. Predicting with little history at a young company
  47. Uplift modelling for targeted interventions
  48. Credit-style scorecard with weight of evidence bins
  49. Selection bias and reject inference
  50. Conformal prediction for guaranteed coverage

Where it starts.

Each project arrives with its purpose, milestones and first tasks written out. Here is how the first three read.

Prediction problem framing document

Most failed models answer a question nobody needed answered at the moment it could be acted on. Before touching data, write down the decision the prediction will change, who will act on it, how many cases they can handle each week and how far ahead the prediction must arrive. A one-page framing agreed with the decision owner prevents months of accurate but useless work.

First tasks

  • Book a 45-minute conversation with the person whose decision the model will inform
  • Write the decision, the actor and the action on one page
  • Note how many predicted cases the team can act on each week
  • Get the decision owner to sign off the framing in writing
  • Revisit the framing document with the decision owner @recurring(yearly)

Target variable and outcome window definition

Converted, defaulted or late all sound obvious until you have to write the query. Deciding exactly what counts as a positive case, over which window after the prediction point, and how to treat cases still in progress, fixes the label every later number depends on. Changing it halfway through means starting again.

First tasks

  • Write the target as a rule someone else could turn into a query
  • Choose the outcome window and note why that length fits the decision
  • Calculate the base rate of the target for each recent quarter
  • Ask an operations colleague to find cases the rule gets wrong

Point-in-time training dataset snapshot

Training tables built from today's version of the data quietly include facts that were unknown at the prediction point: updated addresses, corrected statuses, balances after the event. Rebuilding each row as it looked on its scoring date is the single most effective protection against a model that shines in testing and fails in use.

First tasks

  • List every input source and whether it keeps history or overwrites values
  • Rebuild each training row using values as of its prediction date
  • Drop or reconstruct fields that are overwritten in place
  • Save the build query, row count and build date beside the table

Make it yours.

Install Predictive Modelling, then tell Aurum the date you're aiming for. It checks whether the plan fits the weeks you actually have.

Free · Mac (Apple silicon) and Windows · v1.0.5 · Other platforms

Blueprints are general information to help you organise your own work. They are not medical, financial, legal or other professional advice. See the Terms.