Retrieval Provider Swap Test (QA)

Retrieval Provider Swap Test (QA)

1. 01

   Freeze a golden question set of 40 to 60 prompts drawn from live client work

   Pull real queries from support tickets, sales objections, and past deliverable briefs so the benchmark reflects the work the retainer actually pays for, not synthetic demo questions.

2. 02

   Capture baseline answers, citations, and latency from the incumbent retrieval layer

   Record which source documents each answer cites, the wall-clock response time, and the per-query cost so a later comparison has a fixed reference point.

3. 03

   Re-index the same corpus through a second provider and re-run the identical question set

   A context engine API such as Ragie handles parsing and entity extraction as a managed layer, which makes a side-by-side re-index a same-day task rather than a migration project.

4. 04

   Score answers on citation accuracy, not fluency

   A confident paragraph that cites the wrong clause in a client contract is a liability; grade whether the retrieved passage actually supports the claim before grading tone.

5. 05

   Test the edge cases that break naive pipelines

   Include scanned PDFs, image-heavy decks, and audio transcripts, since multimodal ingestion is where providers diverge most and where client document sets are messiest.

6. 06

   Compare total cost per resolved query, not list price per token

   Fold in re-indexing runs, storage, and the engineering hours spent on connector maintenance; a cheaper per-call rate often loses once ingestion overhead is counted.

7. 07

   Write the swap decision into the client's delivery record with the benchmark attached

   Documenting why the provider changed protects the agency when a client asks why an answer shifted, and it turns an internal QA pass into evidence of diligence.