Eval Baseline Before Client AI Go-Live (Onboarding)

Eval Baseline Before Client AI Go-Live (Onboarding)

1. 01

   Inventory every model call, tool invocation, and retrieval hop the client AI product makes

   Map the full execution graph before scoring anything. Langfuse-style hierarchical traces and Arize-style agent tracing both depend on knowing which spans exist, so an incomplete map produces evaluation gaps that surface as production surprises.

2. 02

   Write 10 to 20 golden test cases drawn from real client conversations, not synthetic prompts

   Pull transcripts from the client's support inbox, sales calls, or prior chatbot logs. Cekura simulates thousands of scenarios with diverse personas, but the highest-signal cases come from interactions the client already had.

3. 03

   Define pass criteria per case with a named owner and a numeric threshold

   For example: refund intent resolved in one turn, 90 percent of the time, owned by the delivery lead. Confident AI and Braintrust both support LLM-as-judge scoring, but a judge without a written threshold produces arguments instead of decisions.

4. 04

   Instrument cost and latency alongside quality on the same dashboard

   Track tokens per session, p95 response time, and cost per resolved conversation. A quality score that improves while cost per resolution triples is not a win, and the client will notice the invoice before the accuracy gain.

5. 05

   Run the suite against the current model version and record the baseline numbers

   Model pricing and behavior shift fast. GPT-6 Sol and Luna launched at 50 percent lower API prices than GPT-5.6, and Claude Opus 5.5 arrived at 40 percent below comparable Fable 5.1 cost, so a baseline dated to last quarter may describe a model the client no longer runs.

6. 06

   Agree on the drift alarm threshold and who gets paged when it trips

   Set a concrete trigger, such as a 5-point drop in pass rate over any 7-day window. Voker and Agnost AI both surface intent shifts and failure patterns from live traffic, but someone on the agency side must own the response.

7. 07

   Deliver a one-page baseline memo to the client before the first production release

   State the pass rate, cost per conversation, and the two failure modes most likely to appear. This memo becomes the reference point for every later retainer conversation about AI performance.