Evaluation Baseline Freeze Before Client Launch (Handoff)

Evaluation Baseline Freeze Before Client Launch (Handoff)

1. 01

   Freeze the scoring rubric and judge configuration at the same commit as the prompt set

   Version the rubric alongside the prompt so a later judge tweak cannot silently change what 'passing' meant on launch day. Confident AI and Braintrust both support experiment snapshots that make this a one-click record rather than a manual export.

2. 02

   Run the frozen suite against the release candidate and store the raw scores, not just the pass rate

   A 92% pass rate hides which 8% failed and why. Keep per-item scores so the next reviewer can see whether failures cluster on a specific intent, language, or tool call.

3. 03

   Record cost and latency percentiles for the same run

   Quality without a p95 latency and per-conversation cost figure is half a baseline. Langfuse dashboards capture both at trace level, which gives the client a defensible number when they ask what the agent costs at volume.

4. 04

   Write the known-limitations list and attach it to the handoff pack

   Name the scenarios the agent handles poorly today, in plain language. Cekura's pre-production scenario runs are useful here because they surface voice-specific failure modes like interruption handling before a client discovers them on a live call.

5. 05

   Define the drift threshold that triggers a re-evaluation

   Pick a concrete trigger, for example a 5-point drop in judge score over a rolling 7-day window or a 20% shift in intent distribution. Arize and Voker both expose intent and correction signals that can feed this threshold.

6. 06

   Assign a named owner on both sides for the first 30 days post-launch

   The agency side owns the trace review; the client side owns the escalation path. Put both names in the handoff document so a production failure has a human attached to it within the hour.

7. 07

   Schedule the 30-day baseline re-run before the kickoff call ends

   Book the calendar slot during handoff, not after. The comparison between launch baseline and day-30 scores is the artifact that justifies the observability line item on the retainer.