Each project arrives with its purpose, milestones and first tasks written out. Here is how the first three read.
Inventory of tick data sets and their sources
Most tick stores grow by accident: a vendor dump on one drive, a capture process on another, a colleague's cleaned copy somewhere in between. Listing every data set with its source, venues, date range, size, format and owner shows what you actually hold, which copies are duplicates and which ones research quietly depends on.
First tasks
- List every drive, bucket and database that holds tick data
- Record source, venues, date range, size and format for each data set
- Ask colleagues which data sets their current research reads
- Mark duplicate and orphaned copies for a consolidation decision
Canonical schema for trades, quotes and book updates
Every vendor and exchange names fields differently, scales prices differently and packs flags in its own way, so research code fills up with per-source special cases. A single canonical schema for trades, top-of-book quotes and book updates, with types, units and nullability fixed, lets every loader translate once and every query read the same columns.
First tasks
- Collect the field lists from every current source side by side
- Decide the price representation and how tick size is stored
- Write the trade, quote and book update layouts with types and units
- Map each source's fields to the canonical columns in one table
Timestamp and time zone policy
A tick can carry three or four times: the exchange's matching time, the feed's send time, your receive time and a vendor's own stamp, each at a different precision and often in local exchange time. Writing down which one is primary, storing it in UTC at nanosecond resolution, and keeping the others as named columns removes the most common silent error in intraday research.
First tasks
- List each source's timestamps with their meaning and precision
- Choose the primary event time and write down why
- Name the secondary time columns and their units
- Add the policy to the schema document and share it with researchers