Failure PatternDecision layer
The Retrieval-Only Trap: Why RAG Tooling Stalls When Agencies Ship Answers Without an Eval Layer
Symptom: Client asks why a chatbot cited a 2023 policy PDF instead of the updated one uploaded last week, and nobody on the delivery team can reproduce the retrieval path that produced the answer. Root cause: Agencies treat retrieval as a plumbing task and skip building their own evaluation layer, so retrieval quality is judged by whoever last looked at a sample output rather than by a repeatable score.
By InnovaAI ResearchPublished Updated
How do you recognize it?
- •Client asks why a chatbot cited a 2023 policy PDF instead of the updated one uploaded last week, and nobody on the delivery team can reproduce the retrieval path that produced the answer.
- •A single RAG vendor's API handles parsing, chunking, and search for every client build, so a pricing change or retrieval quality regression becomes a portfolio-wide incident rather than one project's problem.
- •Demo accuracy looks strong on the three documents used during the pitch, then drops once a client loads 40,000 pages of mixed PDFs, images, and audio transcripts into the same index.
- •Retainer hours get consumed re-tuning chunk sizes and prompts by feel, with no recorded benchmark that shows whether the last change helped or hurt answer grounding.
Why does it happen?
- •Agencies treat retrieval as a plumbing task and skip building their own evaluation layer, so retrieval quality is judged by whoever last looked at a sample output rather than by a repeatable score.
- •Vendor selection happens once during the pitch and never gets revisited, which means the retrieval engine, its pricing, and its parsing behavior are all inherited from a decision made before any client data was loaded.
- •Document ingestion is underestimated: multimodal sources (scanned PDFs, call recordings, image-heavy decks) need different parsing and chunking strategies, and a single default pipeline applied across every client produces uneven grounding.
- •Grounded, source-cited output is sold as a deliverable but the citation itself is never validated, so a plausible-looking reference can point at the wrong passage and pass review.
How do you fix it?
- •Build a 50-question gold set per client with known correct source passages, and score every retrieval change against it before it reaches production.
- •Abstract the retrieval call behind an internal interface so a Ragie index, a Context.dev web-fetch layer, or a self-hosted vector store can be swapped without rewriting prompt logic or the client-facing UI.
- •Log the retrieved chunk IDs and source URLs alongside every generated answer, then spot-check 20 answers per client per week against the original documents.
- •For regulated or audit-heavy client work, route decisions through a rules-plus-retrieval pattern (the approach ai·rete·rag uses) so the verdict is deterministic and the citation explains it rather than determines it.