Agent Chain Failure Containment (QA)

Agent Chain Failure Containment (QA)

1. 01

   Map every agent-to-agent handoff in the client workflow and label each one as reversible or irreversible

   A reversible handoff can be re-run from a saved input; an irreversible one (payment release, client email send, CRM write) needs a human approval gate before it fires.

2. 02

   Assign a named owner and a hard timeout to each agent in the chain

   An agent with no timeout becomes the single point where a stalled run silently burns token budget overnight; cap each hop at a defined wall-clock limit and route timeouts to the owner.

3. 03

   Write the fallback branch before the happy path ships

   For each agent, define what the downstream step receives when the upstream output fails schema validation: a cached prior result, a deterministic template, or a stop-and-escalate signal. Test the fallback branch at least once per release.

4. 04

   Scope every agent's credentials to the minimum its single task requires

   Least-privilege access per request, as seen in tools like Nonce for Gmail and Calendar, limits blast radius when one agent in the chain misbehaves. Audit connected-account permissions quarterly against the task definition.

5. 05

   Run a hallucination propagation test on the full chain, not on individual agents

   Seed a plausible but wrong output at step one and confirm downstream agents flag or halt rather than compounding the error into client-facing copy or reports.

6. 06

   Instrument per-hop logging with input, output, model, token count, and latency

   Without hop-level logs, a failed delivery cannot be attributed to a specific agent, and the retainer conversation becomes a blame exercise instead of a fix.

7. 07

   Set a token spend ceiling per workflow run and alert at 70 percent of the ceiling

   Agentic workflows billed by consumption can erode margin fast; a per-run ceiling catches runaway loops before they reach an invoice.

8. 08

   Rehearse one full chain failure with the delivery team each quarter

   Simulate a mid-chain agent outage and time how long the team takes to identify, isolate, and recover the run. Record the recovery time as the client-facing SLA input.