Retrieval Quality Baseline (Onboarding)

Retrieval Quality Baseline (Onboarding)

1. 01

   Collect 30 to 50 real client questions before any indexing work begins

   Pull them from support tickets, sales call transcripts, and the last two quarters of client email threads. Synthetic question sets hide the messy phrasing that breaks retrieval in production.

2. 02

   Define the pass bar in writing with the client sponsor

   Agree on a citation accuracy threshold (for example, 85% of answers must cite a source the client's own reviewer accepts) and a maximum acceptable latency before the build starts.

3. 03

   Inventory every document source and its access constraints

   List each repository, its owner, refresh cadence, and whether it contains regulated data. A context engine API such as Ragie connects to Google Drive, Notion, and Slack, so connector coverage is rarely the blocker; permissions and retention rules usually are.

4. 04

   Run the same question set through two retrieval providers and record raw scores

   Score each answer for citation correctness, completeness, and refusal behavior. Keep the scoring sheet in the client folder so a later provider swap is a re-run, not a rebuild.

5. 05

   Document chunking and parsing decisions as configuration, not code

   Store chunk size, overlap, and parser settings in a versioned config file. When a client adds scanned PDFs or audio, the change is a config diff the delivery team can review.

6. 06

   Set the evaluation cadence and owner before launch

   Name who re-runs the benchmark monthly and where results are logged. For regulated clients, pair the retrieval layer with a deterministic decision engine such as ai·rete·rag so each verdict links back to the exact rule that fired.

7. 07

   Write the provider swap clause into the statement of work

   State that retrieval vendors are interchangeable components behind the agency's evaluation layer. This keeps pricing negotiations open and protects the retainer if accuracy benchmarks shift.