AI Evaluation Rule: Score the Failure, Not the Average
When an AI agent's aggregate quality score looks healthy, how do I know whether the deployment is actually safe to keep running for a client? Break every aggregate score down to the span, intent, and failure-mode level before you renew or expand an agent deployment, because a 94% average can hide a 100% failure rate on the one intent that touches client money.
By InnovaAI ResearchPublished
“When an AI agent's aggregate quality score looks healthy, how do I know whether the deployment is actually safe to keep running for a client?”
Break every aggregate score down to the span, intent, and failure-mode level before you renew or expand an agent deployment, because a 94% average can hide a 100% failure rate on the one intent that touches client money.
Operators ship on a dashboard headline, treating a high pass rate as a release gate, then discover during a client escalation that the failing intent was never segmented out. The fix is not a better average; it is a failure taxonomy with a named owner and a threshold per intent, reviewed on a fixed cadence before the retainer conversation, not after it.
Averages reward volume: an agent that handles thousands of trivial lookups well will bury the handful of broken intents that generate refund requests and escalations. Production-monitoring services in this category exist precisely because standard eval suites miss friction signals, with Agnost AI surfacing failed intents and repeated retries from real conversations and Arize running span, trace, and session-level evaluations rather than one blended number. The risk is not hypothetical: a six-agent deployment sent 187 messages and leaked health data over 15 days without recovering a £350 refund, and a single hacker used largely autonomous agents to breach more than 100 sites including John Deere and SharkNinja. For an agency, the failure that matters is the one a client can screenshot.
- •A client-facing agent reports a pass rate or satisfaction score above the team's internal threshold
- •The eval suite scores whole conversations or sessions rather than individual spans and tool calls
- •Production traffic includes voice or chat flows where a single bad turn ends the interaction
- •The agent has write access to client systems, forms, or budgets
- •A renewal or scope-expansion conversation is coming up and the agency needs defensible evidence