Bot Behavior Regression Testing (QA)

Bot Behavior Regression Testing (QA)

1. 01

   Freeze the current bot configuration and export a versioned snapshot before any prompt, knowledge base, or guardrail change ships

   Store the snapshot alongside the client's brand guide and escalation rules so a rollback is a one-click restore rather than a rebuild.

2. 02

   Assemble a 25-question regression suite drawn from the client's last 90 days of real support tickets

   Split the suite into 15 factual lookups, 5 pricing or availability questions, and 5 adversarial prompts that try to force off-brand or out-of-scope answers.

3. 03

   Run the suite against the candidate build and score each answer as pass, partial, or fail against a written expected response

   A partial is any answer that is factually correct but omits the next step the customer needs, such as a booking link or a human contact.

4. 04

   Test the human handoff path by triggering escalation keywords and confirming the transcript reaches the client's live inbox within 60 seconds

   Platforms like Social Intents route escalations into Teams, Slack, or Google Chat, so verify the destination channel the client actually staffs, not the one configured at setup.

5. 05

   Check knowledge-grounding behavior by asking three questions whose answers are absent from the uploaded documents

   The bot should say it does not know and offer a handoff. A confident invented answer is a release blocker, not a minor defect.

6. 06

   Log every failure with the exact prompt, the bot response, and the expected response in a shared QA sheet the client can read

   This sheet becomes the evidence pack for the monthly retainer report and shortens the next regression cycle.

7. 07

   Re-run the full suite after fixes and require a 100 percent pass on the adversarial and handoff categories before promoting the build

   Factual lookups can ship at 90 percent pass with a documented watch list; safety and escalation failures cannot.

8. 08

   Archive the passing snapshot, the QA sheet, and the model version used, then schedule the next regression run

   Monthly is the floor for active client bots; weekly applies to any bot handling pricing, scheduling, or payment questions.