Conversation Transcript Review Cadence (QA)

Conversation Transcript Review Cadence (QA)

1. 01

   Pull a fixed sample of closed conversations from every client bot each week

   Twenty to thirty transcripts per client is enough to catch systemic failures without drowning a reviewer. Stratify the sample so it includes at least five conversations that ended in a human handoff and five that ended with no resolution.

2. 02

   Score each transcript against four failure classes: wrong answer, loop, dead end, and tone break

   A loop is any exchange where the visitor repeats the same intent three or more times. A dead end is a session that closes after the bot's final message with no next step offered. Tag every transcript with exactly one primary class so the weekly counts stay comparable.

3. 03

   Check whether the bot cited a source the client would stand behind

   Grounding failures usually surface as confident answers with no traceable document behind them. When a bot answers from general model knowledge rather than the client's knowledge base, log the question and flag the gap for the next intake pass.

4. 04

   Verify that every handoff carried context into the human queue

   The receiving agent should see the visitor's stated intent, the bot's last answer, and any collected contact fields. A handoff that drops the transcript forces the visitor to repeat themselves, which is the single most common complaint in chatbot deployments.

5. 05

   Compare the week's failure counts against the prior four weeks before writing anything up

   A single bad week is noise. Three consecutive weeks of rising loop rates on the same intent is a configuration problem worth a change request. Report the trend line, not the snapshot.

6. 06

   Route each confirmed defect to one of three owners: prompt, knowledge base, or escalation rule

   Prompt defects get fixed in the bot configuration. Knowledge base defects go back to the client for source material. Escalation defects change the handoff logic. Naming the owner in the review note prevents defects from sitting unassigned.

7. 07

   Send the client a two-paragraph summary with the defect count and the fix timeline

   Keep it short: how many conversations were reviewed, how many defects were found, what changes ship this week. Clients on a retainer read the summary; they rarely read the full transcript log.