Retrieval Evaluation Harness (QA)
A sequence with 7 steps: Freeze a 50-query golden set drawn from real client questions before any retrieval vendor is wired in.
By InnovaAI ResearchPublished
What are the steps?
Retrieval Evaluation Harness (QA)
- 01
Freeze a 50-query golden set drawn from real client questions before any retrieval vendor is wired in
Pull the queries from support tickets, sales call transcripts, and the last 90 days of client email threads. Store them in a versioned file so every vendor swap is scored against the same bar.
- 02
Label each query with the exact source document and passage that constitutes a correct answer
Two reviewers label independently, then reconcile disagreements. Unresolved labels get dropped from the set rather than guessed at.
- 03
Run the golden set through the current retrieval layer and record hit rate, mean reciprocal rank, and citation precision
Capture the raw retrieved chunks, not just the final answer, so a wrong answer can be traced to a retrieval miss versus a generation miss.
- 04
Score a challenger vendor on the identical set within the same 48-hour window
Run both vendors against the same document corpus snapshot. A context engine API such as Ragie can be swapped in behind the harness without touching prompt logic, which keeps the comparison clean.
- 05
Separate deterministic decision paths from generative explanation paths when the client workflow is regulated
For underwriting, fraud, or eligibility work, route the verdict through a rule layer and use retrieval only to ground the written rationale. ai·rete·rag is built around this split, and it changes what you measure: rule-fire accuracy on one side, citation fidelity on the other.
- 06
Set a swap threshold in writing before the next renewal conversation
Example: migrate if the challenger beats the incumbent by 8 points of citation precision at equal or lower cost per 1,000 queries. A threshold agreed in advance prevents vendor loyalty from overriding the numbers.
- 07
Log every harness run with corpus version, vendor, model, and date in a client-accessible register
This register becomes the evidence pack when a client asks why an answer changed between quarters, and it is the artifact that justifies a retrieval line item on the retainer.