Evaluation Gate Before Client AI Launch (Delivery)

Evaluation Gate Before Client AI Launch (Delivery)

1. 01

   Freeze the prompt and model configuration under version control before any scoring run

   Record model tier, reasoning effort, and system prompt hash so a later regression can be traced to a specific change rather than guessed at. Model naming now separates size, reasoning effort, and multi-agent design, so pin all three.

2. 02

   Assemble a golden set of 50 to 100 real client inputs drawn from past tickets, briefs, or call transcripts

   Synthetic prompts flatter a system. Real inputs carry the typos, partial context, and edge intents that break agents in front of a client.

3. 03

   Score every golden-set response against three named criteria the client has seen

   Typical trio: factual accuracy, brand-voice adherence, task completion. Use LLM-as-judge for volume and hand-label a 20 percent sample to confirm the judge agrees with humans.

4. 04

   Run adversarial cases covering refusal, prompt injection, and out-of-scope requests

   Include at least one case where the agent is asked to act outside its remit. Agents that verbally decline while still attempting the action are a documented failure mode, not a hypothetical.

5. 05

   Trace one full multi-step workflow end to end and log cost and latency per span

   Hallucinations propagate silently through chained steps, so a passing final answer can hide a broken middle. Capture token spend per span to catch the cost spikes that erode retainer margin.

6. 06

   Set pass thresholds in writing and get client sign-off on the numbers before launch

   Example bar: 95 percent accuracy on the golden set, zero injection escapes, p95 latency under 3 seconds. A signed threshold converts a subjective argument into a delivery gate.

7. 07

   Wire production tracing to the same criteria so launch day is not the end of measurement

   Route live traffic into the evaluation dashboard and alert on drift in any scored dimension. Without a live feed, the pre-launch scorecard decays within weeks.