Retrieval Quality Baseline (QA)

Retrieval Quality Baseline (QA)

1. 01

   Freeze a 200-query golden set drawn from real client support tickets, search logs, and sales objections

   Pull verbatim phrasing, including typos and multi-clause questions. Synthetic queries written by the delivery team flatter every index because they match how engineers think, not how end users type.

2. 02

   Label the correct answer for each query before any index is built

   Two reviewers score independently and reconcile disagreements; anything unresolved gets dropped rather than guessed. This label set becomes the scoring contract for the rest of the engagement.

3. 03

   Record a keyword-search control score as the floor to beat

   Run the same golden set through the client's existing search or a plain full-text pass. Semantic retrieval that cannot beat literal matching on the client's own data is not worth a line item on the retainer.

4. 04

   Score recall at k and mean reciprocal rank on the candidate index

   Report recall@10 and MRR side by side. A system that surfaces the right document at position 9 looks identical to one that surfaces it at position 1 under a single accuracy number, and clients notice the difference in the demo.

5. 05

   Stress the metadata filters with the client's real segmentation logic

   Test tenant isolation, date windows, and permission scopes using production-shaped values. Hybrid dense-sparse setups such as Qdrant's filtered search behave very differently once a restrictive filter removes 95 percent of the corpus.

6. 06

   Time p95 latency under concurrent load, not single-query conditions

   Simulate the peak session count the client expects at launch. Sub-50ms figures quoted for Meilisearch or any other engine mean little once ten simultaneous users hit a cold shard.

7. 07

   Run a data-leak probe across tenants and document the result

   Query for terms that exist only in another client's namespace and confirm zero results. Retrieval bugs that cross tenant boundaries are the failure mode most likely to end a retainer.

8. 08

   Sign off the baseline in writing with the client's product owner before build continues

   Attach the golden set, scores, and latency table to the project record. Re-run the identical set after every embedding model change or reindex so regressions surface in hours rather than in a client complaint.