Failure PatternDecision layer
The Prototype-to-Production Gap: Why Vector Databases Stall at Client Scale
Symptom: Semantic search returns relevant results on a 5,000-document demo corpus, then recall drops sharply once a client's full archive of 2 million records is indexed. Root cause: Pilots are scoped to prove that semantic search works at all, not to prove it holds at the data volumes and concurrency the retainer actually requires.
By InnovaAI ResearchPublished
How do you recognize it?
- •Semantic search returns relevant results on a 5,000-document demo corpus, then recall drops sharply once a client's full archive of 2 million records is indexed
- •Query latency climbs from under 100ms in testing to several seconds under concurrent load, and the client notices before the agency does
- •Recall and precision were never measured, so nobody can prove whether the retrieval layer or the language model is producing wrong answers
- •The agency cannot state a per-client monthly cost for the retrieval layer because nobody tracked index size, query volume, or replica count
- •Re-indexing after a schema or embedding model change takes hours and requires a maintenance window the client never agreed to
Why does it happen?
- •Pilots are scoped to prove that semantic search works at all, not to prove it holds at the data volumes and concurrency the retainer actually requires
- •Retrieval quality is treated as a model problem when it is an indexing and filtering problem: chunking strategy, metadata filters, and hybrid dense-sparse search decide most of the outcome
- •Managed and self-hosted options carry different cost curves, and agencies pick one before mapping which curve fits the client's query pattern and data residency needs
- •No one owns the retrieval layer after launch, so index hygiene, embedding versioning, and capacity planning fall to whoever is on call that week
How do you fix it?
- •Run a load test at 10x the pilot corpus size before signing any statement of work that promises semantic search, and record recall, precision, and p95 latency as contract baselines
- •Instrument the retrieval layer separately from the generation layer so a wrong answer can be attributed to indexing, filtering, or the model within one debugging session
- •Write a one-page retrieval runbook per client covering re-index triggers, embedding model version, and the rollback path, then attach it to the retainer deliverable list
- •Model three cost scenarios (pilot, steady state, peak season) across one managed platform and one self-hosted engine before committing the client to either deployment model