Gateway Failover Drill (QA)

Gateway Failover Drill (QA)

1. 01

   Freeze the production base URL and record it in the client delivery runbook

   Write down the exact endpoint, model identifier, and SDK version each client integration points at. Without this baseline, a swap test proves nothing because nobody can confirm what changed.

2. 02

   Stand up a second OpenAI-compatible endpoint on a separate host and mirror the model list

   Pair a deterministic-serving gateway such as TokenDelivery.ai with a fixed-pass host like endlesstokens so the fallback differs in billing model as well as infrastructure. Two hosts on the same upstream GPU pool is not a fallback.

3. 03

   Replay a frozen prompt set against both endpoints and diff the raw responses

   Use 20 to 50 real client prompts captured from production traffic, including at least one tool-call and one structured-output request. Byte-level comparison matters where reproducibility is contractual; where it is not, compare field-level output shape instead.

4. 04

   Time the swap: change base URL and key only, then confirm zero code edits

   The whole premise of the category is that a model change is a config change. If the drill requires touching client-side code, the integration was never portable and the retainer is exposed.

5. 05

   Measure latency and cost per 1,000 requests on the fallback path

   Record p50 and p95 latency plus effective cost under each host's pricing shape (per-token versus time-based pass). A pass-based host can be cheaper for bursty batch jobs and more expensive for steady low-volume client workloads.

6. 06

   Run the drill on a scheduled cadence and log results against the runbook

   Quarterly is the floor; monthly for any client whose delivery halts if inference stops. Log the date, endpoints tested, diff outcome, and any config drift found.

7. 07

   Escalate any failed drill to the account owner before the next client status call

   A failed failover is a delivery risk, not an engineering ticket. Surface it with the remediation date so the client hears it from the agency first.