Model Routing and Fallback Map (Delivery)

Model Routing and Fallback Map (Delivery)

1. 01

   Inventory every client workflow that calls a foundation model and tag each by latency and context demand

   Split the list into real-time paths (voice agents, chat, browser-use) and batch paths (reporting, summarisation, asset generation). A 1M-token context job and a sub-second voice turn should never share the same routing rule.

2. 02

   Assign a primary model per workflow using measured price per million tokens, not list price

   Price the actual input/output mix for each workflow. A cheaper per-token model with heavy retries or long prompts can cost more than a premium tier at the volumes a retainer actually runs.

3. 03

   Name a secondary provider for every primary and confirm request-format compatibility

   OpenAI- and Anthropic-compatible endpoints make swap-in realistic; DeepSeek and Inception both expose compatible request formats, so a fallback can be wired without rewriting client code.

4. 04

   Set the fallback trigger thresholds in writing before launch

   Define error-rate, p95 latency and rate-limit ceilings that force a switch. Tiers and limits change per account, so thresholds must be numbers the delivery team can watch, not a judgement call at 2am.

5. 05

   Run a side-by-side quality test on the client's own inputs, including edge cases

   Technical, multilingual and adversarial prompts expose routing behaviour that generic benchmarks miss. Mixture-of-Experts routing decisions inside a model change output quality and inference cost per token, so test with the content types the client actually ships.

6. 06

   Document data-retention and training-use terms for each provider in the client file

   Record whether prompts and outputs are retained, for how long, and whether they train on client data. Private deployment options matter when the client's legal team asks, and the answer should already be on file.

7. 07

   Log the routing map in the delivery runbook with an owner and a review date

   One named person owns the map and re-checks it at each model release cycle. Without an owner, fallbacks rot and the first sign of failure is a client-facing outage.