Model Evaluation and Procurement Gate (Onboarding)

Model Evaluation and Procurement Gate (Onboarding)

1. 01

   Define the client deliverable before shortlisting any model

   Write one sentence naming the output (voice agent, RAG assistant, ad creative batch) and the volume per month. A 1M-token context window matters for document work and is wasted spend on a 200-word caption generator.

2. 02

   Score candidates on the five procurement dimensions, not on benchmark charts

   Model quality on your own inputs, context length, p95 latency, price per million tokens, and data-retention terms. Run each candidate against 20 to 30 real client samples, including the messy multilingual and edge-case ones.

3. 03

   Test routing behavior under load before committing volume

   Mixture-of-Experts routers pick expert sub-networks per token, so Top-K routing can spike latency when popular experts saturate. Replay a peak-hour traffic pattern against each shortlisted endpoint and record tail latency, not just averages.

4. 04

   Map a primary and a fallback endpoint with compatible request formats

   OpenAI- and Anthropic-compatible APIs let you swap providers without rewriting client code. Confirm the fallback returns the same JSON schema and tool-call structure so a failover does not break the delivery pipeline mid-campaign.

5. 05

   Document who controls the meter, the gate, and the referee for each vendor

   Record pricing authority, feature-access control, and dispute resolution in one table. Forrester flags that vendors holding all three roles turn a routine price change into an unbudgeted client delivery risk.

6. 06

   Confirm data-retention and residency terms in writing

   Ask for the retention window, training-on-customer-data policy, and whether private VPC or on-premises deployment is available. Client contracts with regulated industries usually require this before the first API call.

7. 07

   Price the retainer line with a token budget and a 30 percent buffer

   Estimate monthly tokens from the pilot volume, multiply by the published per-million rate, and add headroom for retries and agent loops. Present it as a named line item so scope creep is visible on the invoice.

8. 08

   Label the engagement stage and attach the matching deliverable

   A proof of concept answers feasibility, a pilot answers production readiness, and an MVP answers whether it ships. Skipping the pilot stage is the most common way an agency AI project overruns its budget.