Rollback Drill and Incident Rehearsal (QA)

Rollback Drill and Incident Rehearsal (QA)

1. 01

   Pick one live client deployment and freeze its release window for the drill

   Choose a low-traffic property with a known-good prior build so a failed rehearsal costs nothing beyond a 30-minute window. Notify the account manager before touching production.

2. 02

   Reproduce the last production change from the repository, not from memory

   Trigger the same commit through the pipeline that shipped it (DeployHQ, Railway, or a YAML manifest in systemg) so the drill tests the real path rather than a hand-run script.

3. 03

   Break the deploy on purpose and time the detection

   Introduce a failing build step or a bad environment variable, then record how long until the pipeline halts and who gets the alert. Anything over five minutes of silent failure is a gap to log.

4. 04

   Execute the rollback and confirm the previous version serves traffic

   One-click rollback exists on several platforms, but verify the database migration state separately. A code rollback against an un-reverted schema is the most common cause of a second outage.

5. 05

   Walk the client-facing communication path with a real message

   Draft the status note the client would receive, including what broke, what was reverted, and the next update time. Route it through whoever actually holds the client relationship, not the engineer on call.

6. 06

   Record time-to-detect and time-to-restore in the client's delivery file

   Two numbers per drill, tracked across quarters. These become the evidence base when a retainer renewal conversation turns to reliability.

7. 07

   Assign one owner and one due date per gap found, then schedule the next drill

   Gaps without an owner do not close. Book the following drill before the retro ends so the cadence survives a busy delivery month.