LLM Evaluation & RAG Data

A corpus you are allowed to index, real evaluation questions with reference answers, a harness and regression gate on every change, and a weekly habit of reading what the assistant actually says.

Free · Mac (Apple silicon) and Windows · v1.0.5 · Other platforms

Then open the Marketplace in the app, search for LLM Evaluation & RAG Data, and click Install. It goes into your own files, and works offline from then on.

What it installs.

1

Pillar

1

Area

50

Projects

206

Starter tasks

📊 Data & Analytics

LLM Evaluation & RAG Data

Preparing document collections for retrieval, building evaluation sets and measuring the quality of language model answers over time.

  1. Definition of a good answer for the assistant
  2. Source document inventory for retrieval
  3. Corpus permissions, licensing and sensitive content check
  4. First evaluation set of real user questions
  5. Reference answers and supporting passages
  6. Evaluation harness that runs on every change
  7. Retrieval quality baseline with recall at k
  8. Answer scoring rubric with a graded scale
  9. Experiment log for prompts, models and retrieval settings
  10. Weekly production answer sample review
  11. Corpus refresh and re-indexing schedule
  12. Stale and superseded document sweep
  13. Regression gate for prompt and model changes
  14. LLM judge calibration against human labels
  15. User feedback triage for flagged answers
  16. Quarterly evaluation set refresh
  17. Monthly quality, cost and latency report
  18. Model provider version and deprecation watch
  19. Retrieval metrics study with recall, MRR and nDCG
  20. LLM-as-judge methods and known biases
  21. Embeddings and vector search fundamentals
  22. Confidence intervals for small evaluation sets
  23. Annotation guidelines and reviewer agreement
  24. Hallucination and faithfulness error taxonomy
  25. Prompt injection through retrieved documents
  26. Document parsing pipeline for PDFs and tables
  27. Chunking strategy comparison
  28. Embedding model selection on your own data
  29. Hybrid search and reranker trial
  30. Document metadata and retrieval filters
  31. Citations and source attribution in answers
  32. Abstention when retrieval comes back weak
  33. Vector database or search engine choice
  34. Near-duplicate removal in the corpus
  35. Pre-launch evaluation sign-off
  36. Red team session before launch
  37. Language model version migration evaluation
  38. Internal pilot with twenty users
  39. Post-incident review of a harmful or wrong answer
  40. Solo engineer adding RAG to an existing product
  41. Research benchmark dataset for publication
  42. Expert review panel for a regulated domain assistant
  43. Multilingual corpus and evaluation
  44. Inheriting a RAG system with no evaluations
  45. Knowledge assistant handover to a support team
  46. Synthetic question generation with quality filters
  47. Agent and tool-use trajectory evaluation
  48. Long context versus retrieval comparison
  49. Benchmark contamination checks
  50. Embedding fine-tuning with hard negatives

Where it starts.

Each project arrives with its purpose, milestones and first tasks written out. Here is how the first three read.

Definition of a good answer for the assistant

Before anyone builds an evaluation set, the team needs to agree what the assistant is for, who asks it questions and what a good answer looks like for them. A one-page definition covering the main question types, the audience, the tone, when the assistant should decline and which failures are unacceptable turns vague complaints that it feels worse into things you can score.

First tasks

  • Pull fifty real questions from logs, tickets or chat history to see what people ask
  • Group the questions into five to eight types with one example each
  • Write the list of failures that would make an answer unacceptable
  • Review the one-page definition with the product owner and a domain expert

Source document inventory for retrieval

Retrieval quality is capped by the documents you index, and most teams start by pointing a loader at one shared drive without knowing what else holds the real answers. An inventory of every candidate source, its format, size, owner, update frequency and authority tells you what to index first and what to leave out.

First tasks

  • Ask three people who answer these questions today where they look first
  • List each source with its format, document count, owner and change frequency
  • Agree an authority level for each source with the content owners
  • Pick the first-wave sources and write one line on why each was chosen
  • Review the source inventory for new or retired systems @recurring(quarterly)

Corpus permissions, licensing and sensitive content check

Indexing a document makes its content reachable by anyone who can query the assistant, so a file that was restricted in its home system can leak through an answer. Checking licences, access groups and personal or confidential content before ingestion, and carrying permissions into the index, prevents the most serious class of retrieval incident.

First tasks

  • List the access groups that can see each first-wave source today
  • Run a sensitive content scan on a sample of documents from every source
  • Write the exclusion rules into the ingestion configuration
  • Book a review of the approach with the security or privacy owner
  • Re-check access groups for the indexed sources @recurring(quarterly)

Make it yours.

Install LLM Evaluation & RAG Data, then tell Aurum the date you're aiming for. It checks whether the plan fits the weeks you actually have.

Free · Mac (Apple silicon) and Windows · v1.0.5 · Other platforms

Blueprints are general information to help you organise your own work. They are not medical, financial, legal or other professional advice. See the Terms.