Provider Swap Readiness Review (Retention)

Provider Swap Readiness Review (Retention)

1. 01

   Freeze the current retrieval configuration and record its exact settings

   Capture chunk sizes, embedding model versions, index types, and connector credentials in a dated snapshot so any regression can be traced to a specific change rather than blamed on the vendor.

2. 02

   Re-run the client's standing query set against the incumbent engine and log precision, recall, and citation accuracy

   Use the same 40 to 60 queries the client's users actually type, not synthetic benchmarks, because retrieval quality on real phrasing is what shows up in client satisfaction scores.

3. 03

   Stand up a parallel index on a second provider and ingest the same corpus

   A context engine API such as Ragie can be provisioned alongside the incumbent in days, letting the agency compare parsing fidelity on PDFs, images, and audio without disrupting live delivery.

4. 04

   Score both engines on the same rubric and document where each wins

   Record per-query deltas rather than a single aggregate score; a provider that loses on average may still win on the two document types that matter most to this client.

5. 05

   Model the cost delta at the client's actual monthly query volume

   Include ingestion, re-indexing, and connector sync charges, since per-query pricing alone understates the true cost of a migration.

6. 06

   Present the comparison to the client with a recommendation and a rollback path

   Frame the decision around retrieval accuracy and auditability, not vendor preference, and confirm the incumbent configuration stays warm for at least one billing cycle after any switch.

7. 07

   Update the agency's internal evaluation layer so the next swap takes hours, not weeks

   Abstracting the retrieval call behind a shared interface means the evaluation harness, not the vendor SDK, becomes the asset the agency owns and reuses across retainers.