Retrieval Evaluation Harness Setup (Delivery)

Retrieval Evaluation Harness Setup (Delivery)

1. 01

   Freeze a 40 to 60 question gold set from real client queries before any provider is wired in

   Pull the questions from support tickets, sales calls, and the client's own FAQ. Each item needs a known correct answer and the source document that contains it, so scoring stays objective across vendors.

2. 02

   Define the four metrics the harness will score on every run

   Use answer accuracy, citation precision, p95 latency, and cost per 1,000 queries. Anything the client cannot see in a monthly report does not belong in the harness.

3. 03

   Run the gold set against at least two retrieval engines in parallel

   A managed context engine such as Ragie and a web data API such as Context.dev will surface different failure modes: one on document parsing, the other on live page freshness. Two data points beat one vendor's benchmark page.

4. 04

   Log every failure with the retrieved chunk, not just the final answer

   Chunk-level logs show whether the miss came from ingestion, ranking, or the prompt layer. Without that split, teams blame the vendor for a chunking decision they made themselves.

5. 05

   Set a swap threshold in writing before the client sees the first demo

   Example: if accuracy drops more than 8 points or cost per query rises above $0.004 for two consecutive monthly runs, the account moves to the alternate provider. Thresholds agreed in advance turn a renegotiation into a routine maintenance task.

6. 06

   Re-run the harness monthly and archive the scores alongside the retainer deliverable

   A dated score history is the evidence an agency needs when a client asks why the AI answers changed, and it is the same file that justifies a provider switch without a scope conversation.