Model Swap Readiness Review (QA)

Model Swap Readiness Review (QA)

1. 01

   Freeze the current model version and record it against every live client deliverable

   Log the exact model name and version behind each retainer output (copy drafts, code branches, research briefs) so a swap can be traced back to a known baseline.

2. 02

   Re-run a fixed 20-prompt regression set on the candidate model before any team-wide switch

   Pull prompts from real client work across technical, multilingual, and edge-case inputs; routing and architecture changes inside a model can shift output quality and inference cost without any visible interface change.

3. 03

   Score outputs blind against the incumbent on accuracy, tone, and revision count

   Two reviewers who did not write the prompts grade each pair; track how many edits a draft needs before it is client-ready, not how fluent it reads.

4. 04

   Test prompt ownership framing on any coding or refactor task

   Anthropic testing on Claude Fable 5.2 found the model preserves existing code structure when prompts omit ownership context, so state explicitly that structure should be improved and confirm the candidate behaves the same way.

5. 05

   Verify data controls and retention terms for every client tier before rollout

   Confirm whether uploaded client files train future models, where data is stored, and which admin toggles disable retention; document the answer per client contract.

6. 06

   Model the cost per seat at realistic agency usage, including peak-hour throttling

   Estimate tokens per active seat per week from last month's logs, then price the candidate at list rates plus any planned increases; a headline price that doubles on a known date belongs in the client margin sheet.

7. 07

   Stage the swap on one delivery pod for two weeks and compare cycle time

   Measure time from brief to first client-ready draft on the pilot pod against a control pod still on the incumbent model.

8. 08

   Publish the swap decision, rollback trigger, and owner in the agency wiki

   Name the person who can revert the standard, the metric threshold that forces a revert, and the date of the next scheduled review.