Each project arrives with its purpose, milestones and first tasks written out. Here is how the first three read.
Definition of a good answer for the assistant
Before anyone builds an evaluation set, the team needs to agree what the assistant is for, who asks it questions and what a good answer looks like for them. A one-page definition covering the main question types, the audience, the tone, when the assistant should decline and which failures are unacceptable turns vague complaints that it feels worse into things you can score.
First tasks
- Pull fifty real questions from logs, tickets or chat history to see what people ask
- Group the questions into five to eight types with one example each
- Write the list of failures that would make an answer unacceptable
- Review the one-page definition with the product owner and a domain expert
Source document inventory for retrieval
Retrieval quality is capped by the documents you index, and most teams start by pointing a loader at one shared drive without knowing what else holds the real answers. An inventory of every candidate source, its format, size, owner, update frequency and authority tells you what to index first and what to leave out.
First tasks
- Ask three people who answer these questions today where they look first
- List each source with its format, document count, owner and change frequency
- Agree an authority level for each source with the content owners
- Pick the first-wave sources and write one line on why each was chosen
- Review the source inventory for new or retired systems @recurring(quarterly)
Corpus permissions, licensing and sensitive content check
Indexing a document makes its content reachable by anyone who can query the assistant, so a file that was restricted in its home system can leak through an answer. Checking licences, access groups and personal or confidential content before ingestion, and carrying permissions into the index, prevents the most serious class of retrieval incident.
First tasks
- List the access groups that can see each first-wave source today
- Run a sensitive content scan on a sample of documents from every source
- Write the exclusion rules into the ingestion configuration
- Book a review of the approach with the security or privacy owner
- Re-check access groups for the indexed sources @recurring(quarterly)