Retrieval Provider Swap Test (QA)

Retrieval Provider Swap Test (QA)

1. 01

   Freeze a 50-question golden set drawn from live client queries before any provider change

   Pull real questions from the last 90 days of client support tickets and prompt logs, then store expected answers with the exact source passage that should be cited. Fifty is enough to expose retrieval drift without burning a week of delivery time.

2. 02

   Score the incumbent engine on citation accuracy and answer recall as a baseline

   Record two numbers per question: did the answer cite the correct document, and did it contain the fact at all. Anything below 85 percent citation accuracy on document-heavy work is already a client-facing risk.

3. 03

   Run the same set through two alternative engines and diff the failures question by question

   A managed context API such as Ragie, a web data layer like Context.dev, and a verification pass through CiteLLM will fail on different inputs. The diff, not the aggregate score, tells you which failure modes your client's corpus actually triggers.

4. 04

   Price the swap at realistic token and page volume, not at list tier

   Recompute monthly cost using the client's actual document count and query volume. Token-based billing on agentic workflows can erode retainer margin quietly, so model the worst-case month before recommending a move.

5. 05

   Confirm the verification layer survives the swap unchanged

   If source highlighting is handled by a separate verification tool, the swap should not touch the client's audit trail. Test that every claim still links to the highlighted passage in the original PDF after the engine changes.

6. 06

   Document the abstraction boundary so the next swap takes hours, not weeks

   Keep ingestion, retrieval, and citation rendering behind your own interface. The point of the test is to prove the boundary holds, not to crown a permanent winner.

7. 07

   Present the swap decision to the client with the before-and-after numbers attached

   Show baseline accuracy, post-swap accuracy, and the cost delta on one page. Clients approve provider changes when the evidence is a scorecard rather than an opinion.