Each project arrives with its purpose, milestones and first tasks written out. Here is how the first three read.
Source system and pipeline inventory
This is a team plan, best installed into a shared workspace so everyone with a role in it can see and own their tasks. Before changing anything, you need one list of every system that feeds the warehouse: SaaS apps, production databases, spreadsheets someone uploads by hand and vendor file drops.
First tasks
- Export the list of jobs from the orchestrator and the connector tool
- List every warehouse schema that receives raw loads
- Reconcile the lists and record each source with its owner and schedule
- Flag jobs that have no owner or no downstream table reading them
- Add new sources to the inventory and mark retired ones @recurring(quarterly)
Freshness targets agreed with data consumers
Teams rarely say how fresh their data needs to be until it is late, and then every pipeline is urgent. Agreeing a freshness target per source with the people who use it, such as hourly for orders and daily for HR, tells you where to spend engineering effort and gives alerts a threshold that means something.
First tasks
- List the five reports people complain about most when they are late
- Ask each consumer what decision the data supports and by what time
- Assign every source a freshness tier and a target in hours
- Publish the targets next to the pipeline inventory
- Review freshness tiers with consumers at the start of the year @recurring(yearly)
Raw landing schema conventions
Raw tables are the evidence trail for everything built on top, so they should look predictable and never be edited by hand. Fixing the conventions now, one schema per source, original column names kept, a load timestamp and batch id on every row, saves analysts from guessing what a table is and makes reloads safe.
First tasks
- Write the naming rule for raw schemas and tables
- Define the load metadata columns every pipeline must add
- Remove human write access to raw schemas in the warehouse
- List raw tables that break the convention and decide which to fix