Retrieval Swap Readiness Review (Retention)

Retrieval Swap Readiness Review (Retention)

1. 01

   Freeze a 40-query golden set per client account before the next renewal conversation

   Pull the queries that actually drive client deliverables (policy lookups, product spec questions, contract clause retrieval) rather than synthetic benchmarks. Forty queries is enough to expose ranking regressions without turning the review into a research project.

2. 02

   Score the incumbent retrieval layer on hit rate, citation precision, and p95 latency

   Record three numbers per query: did the correct passage appear in the top 5, was the cited span the one a human would cite, and how long did the call take. Ragie's context engine API returns parsed chunks and entity metadata, so citation precision can be graded against the returned span rather than the whole document.

3. 03

   Run the same golden set against one alternative provider on a shadow index

   Keep the shadow index fed from the same source documents so the comparison isolates retrieval quality from ingestion differences. For regulated accounts, ai·rete·rag's pattern of using rules as retrieval filters is worth testing because deterministic rule gating changes which passages are even eligible.

4. 04

   Price the swap in engineering hours, not just API spend

   Estimate re-indexing time, connector rewrites, and prompt adjustments. A provider that saves $400 a month but costs 30 engineering hours to migrate is a 6-month payback at typical agency blended rates.

5. 05

   Confirm the abstraction layer actually holds by swapping one non-critical client first

   If the evaluation harness, prompt templates, and citation formatter are genuinely provider-agnostic, a single low-stakes account should migrate in under a day. If it takes a week, the abstraction is nominal and the lock-in risk is real.

6. 06

   Document the swap decision and its evidence in the client's delivery file

   Note the golden-set scores, the cost delta, and the migration effort. When a client asks why retrieval behavior changed mid-retainer, this file is the answer, and it also protects the agency if accuracy complaints surface later.

7. 07

   Set the next review date against the client's contract renewal, not the calendar quarter

   Retrieval pricing and model quality move faster than annual planning cycles. Tying the review to renewal keeps the swap decision attached to a moment when the agency can actually renegotiate scope or rates.