Retrieval Quality Benchmarking Protocol (QA)

Retrieval Quality Benchmarking Protocol (QA)

1. 01

   Define a golden question set from live client intents

   Pull 20 to 50 real queries from support tickets, sales conversations, and product docs, then group them by difficulty and source type.

2. 02

   Establish ground-truth answers for each question

   Have a subject matter expert write the ideal response and tag the exact source passages that support it, so scoring stays objective.

3. 03

   Run the same questions against your current RAG stack

   Capture retrieval results and generated answers in a spreadsheet, noting latency, citation accuracy, and whether the final answer matches the ground truth.

4. 04

   Score each response on citation fidelity and completeness

   Use a simple 0-2 scale per question: 0 for hallucinated or missing citations, 1 for partial support, 2 for fully grounded answers.

5. 05

   Compare scores against a second RAG provider or a baseline model

   If you only use one vendor, test with a different chunking strategy or retrieval mode to isolate whether the gap is in the tool or your configuration.

6. 06

   Document the failure patterns you observe

   Note whether errors come from missing chunks, poor ranking, or prompt mishandling, then share the log with the delivery team so fixes target the real cause.

7. 07

   Re-run the benchmark quarterly and after any vendor change

   Retrieval quality drifts as models and indexes update, so schedule a recurring review and keep the golden set versioned to track improvements over time.