Conversation Transcript Review (QA)

Conversation Transcript Review (QA)

1. 01

   Pull a fixed weekly sample of at least 50 closed conversations per client deployment

   Stratify the sample across channels and time of day so overnight and weekend traffic is not invisible. A 50-conversation floor gives enough signal to spot a broken intent without reading thousands of logs.

2. 02

   Score each transcript against four failure classes: wrong answer, no answer, loop, and premature handoff

   Tag the specific turn where the failure starts, not just the conversation. Turn-level tagging is what makes the fix actionable for whoever owns the knowledge base.

3. 03

   Check every escalation for whether the human agent received context or started cold

   A handoff that forces the customer to repeat themselves reads as a failure even when the bot answered correctly. Confirm the transcript, contact record, and prior turns travel with the escalation.

4. 04

   Compare containment rate against the baseline captured at go-live, not against vendor marketing figures

   Track the delta on the client's own traffic. A platform that advertises 80 percent containment can still sit at 40 percent on a niche B2B support queue with long-tail questions.

5. 05

   Log every unanswered question into a gap register with the exact customer phrasing

   Feed the register into the next knowledge-base update cycle. Verbatim phrasing matters because retrieval quality depends on matching how customers actually ask, not how the client writes documentation.

6. 06

   Flag any transcript where the bot gave pricing, legal, or medical guidance outside approved scope

   Escalate to the client contact within one business day and document the response. Unapproved commitments made by a chatbot become the agency's problem at renewal.

7. 07

   Deliver a one-page QA summary to the client with three fixes ranked by conversation volume affected

   Tie each fix to a retainer deliverable so the review reads as managed optimization rather than a bug report. This is the artifact that justifies the ongoing optimization line item.