Failure PatternDecision layer
The Single-Vendor Retrieval Trap: Why RAG Tooling Lock-In Stalls Agency Delivery
Symptom: Retrieval accuracy on a client corpus drops after a vendor-side index or embedding change, and nobody on the delivery team can explain why the same query now returns different chunks. Root cause: The retrieval layer is treated as plumbing rather than as a product surface, so no evaluation harness is built before the first client deployment and there is no baseline to compare vendors against.
By InnovaAI ResearchPublished
How do you recognize it?
- •Retrieval accuracy on a client corpus drops after a vendor-side index or embedding change, and nobody on the delivery team can explain why the same query now returns different chunks.
- •Agencies quote RAG work at a fixed retainer, then discover that per-query or per-document pricing on the managed layer makes the margin disappear once a client's corpus passes a few thousand files.
- •Two clients with near-identical document workflows get materially different answer quality because one was built on Ragie and the other on a different context engine, yet the agency has no shared benchmark to prove which is better.
- •When a client asks for a regulated audit trail, the team cannot show which retrieved passages produced a given answer because the retrieval layer logs are not exported or retained.
- •Swapping providers during a pilot takes weeks instead of days because chunking, entity extraction, and connector logic were written directly against one vendor's API shape.
Why does it happen?
- •The retrieval layer is treated as plumbing rather than as a product surface, so no evaluation harness is built before the first client deployment and there is no baseline to compare vendors against.
- •Connector and ingestion logic is written against vendor-specific SDK calls, which means the cost of switching is measured in engineering days rather than in configuration changes.
- •Pricing models in this category mix per-document, per-query, and per-seat components, and agencies rarely model the break-even point against a client's actual query volume before signing a retainer.
- •Regulated clients need deterministic, explainable decisions with a full audit trail, and general-purpose context engines do not provide that by default, so teams either over-promise or bolt on a rules layer late.
How do you fix it?
- •Build a 50-question golden set per client corpus and run it against at least two retrieval providers before committing to a retainer, recording precision and citation accuracy side by side.
- •Wrap the vendor call behind an internal retrieval interface so chunking, ranking, and connector logic live in agency-owned code and a provider swap becomes a config change.
- •Model the per-query and per-document cost at the client's projected 12-month volume, and put a volume-based repricing clause in the statement of work.
- •For regulated engagements, pair the retrieval layer with a deterministic rules engine such as ai·rete·rag so each verdict links back to the exact rule and passage that produced it.