Retrieval Quality Gate Before Client-Facing Launch (QA)

Retrieval Quality Gate Before Client-Facing Launch (QA)

1. 01

   Freeze a 50-query golden set drawn from real client questions before tuning anything

   Pull the queries from support tickets, sales call transcripts, and search logs. A set built from synthetic prompts will pass while production traffic still fails.

2. 02

   Score baseline recall at k=10 and record the number in the project log

   Recall at 10 is the cheapest signal of whether the index actually holds the answer. Without a written baseline, every later change becomes an argument instead of a measurement.

3. 03

   Test metadata filters against the client's real segmentation, not a demo subset

   Qdrant and Meilisearch both expose rich filtering, but filters that work on 200 documents can collapse on 2 million. Run the filter with the largest tenant or region the client has.

4. 04

   Verify embedding model version pinning across ingest and query paths

   A mismatch between the model that wrote the vectors and the model that reads them produces plausible but wrong results. Pin the version in config and note it in the handoff doc.

5. 05

   Run the same golden set through a second retrieval path and compare overlap

   Query a hybrid setup such as Weaviate or a full-text fallback in Meilisearch, then measure how many top-10 results agree. Low overlap means one path is carrying the system and the other is decoration.

6. 06

   Confirm the client's data residency and training-pool terms in writing

   Forrester's September 2026 position is that private deployments beat shared public models for B2B marketing because shared access erases differentiation. Get the vendor's answer on whether client embeddings leave the tenant.

7. 07

   Log latency at p95 under the client's expected concurrent load

   A demo at 40ms can become 900ms when a campaign sends 3,000 users to the assistant at once. Load-test before launch, not after the first angry email.

8. 08

   Sign off with a one-page results sheet the account lead can forward

   Include the golden set size, baseline and final recall, p95 latency, and any known failure queries. Clients accept documented gaps; they do not accept surprises in month two.