Most RAG systems fail in the same predictable ways. This is a working architecture, the failure modes we have seen in production, and the design decisions that separate a demo from something you can put in front of a board.
The three failures we see most
Nearly every broken RAG system we are asked to review fails on one of three axes:
- Retrieval that returns plausible-but-wrong context — the model answers confidently from the wrong chunk.
- No grounding contract — the model is free to answer from its own priors when retrieval comes back thin.
- No confidence signal — every answer looks equally authoritative, so users can’t tell a solid finding from a guess.
The grounding contract
The single most important design decision is to forbid the model from answering outside the retrieved context. In practice that means a system prompt that treats retrieved passages as the only source of truth:
def build_prompt(query: str, chunks: list[Chunk]) -> list[dict]:
context = "\n\n".join(f"[{c.id}] {c.text}" for c in chunks)
system = (
"Answer ONLY from the numbered context below. "
"Every claim must cite a source id like [3]. "
"If the context does not answer the question, say so — do not guess."
)
return [
{"role": "system", "content": system},
{"role": "user", "content": f"Context:\n{context}\n\nQuestion: {query}"},
]
That last instruction — say so, do not guess — is what converts a confident hallucination into an honest “the data doesn’t cover this.”
Confidence, made legible
Retrieval scores are a signal, not a verdict. We map them to labels a non-technical reader can act on:
| Retrieval signal | Label | What it tells the reader |
|---|---|---|
| Strong match, multiple sources agree | Established | Act on it |
| Consistent but thinner evidence | Suggestive | Plan around it |
| Weak or single-source | Exploratory | Watch it |
A finding a reader can trace to its source is worth more than a summary they have to take on faith.
Retrieval is only as good as the question you let the model ask
The grounding contract stops the model from inventing answers. But it can’t rescue a system that retrieves the wrong context in the first place — and this is where most of the real work lives. A model constrained to answer only from what it retrieved will still be wrong if what it retrieved is wrong.
Two things dominate retrieval quality. The first is chunking — how the source material is split before it’s embedded. Chunks that are too large bury the relevant sentence in noise and dilute the match; chunks that are too small lose the context that made the sentence meaningful. There’s no universal right answer, but there is a right method: chunk along the document’s own structure (sections, entries, records) rather than by arbitrary character counts, so each chunk is a coherent unit of meaning. The second is query handling. The user’s raw question is often not the best retrieval query — it may be underspecified, or phrased in language that doesn’t match the source. Rewriting or expanding the query before retrieval, and routing different kinds of questions to different sources, does more for answer quality than swapping in a bigger model ever will.
The principle underneath both: retrieval is only as good as the question you let the model ask. Spend your effort there, not on the generation step.
When the honest answer is “route, don’t retrieve”
Free-text retrieval is the right tool when the answer genuinely lives in prose — documents, notes, policies, guidance. But a large class of questions that look like retrieval problems are actually lookup problems in disguise. “What was the testing rate last quarter?” is not a question you want answered by retrieving text that mentions testing rates and hoping the model reads them correctly. It’s a question that should route to a defined metric, computed by verified code.
Knowing which mode a question needs — retrieve prose, or route to a governed calculation — is itself a design decision, and getting it wrong is a common source of confidently wrong numbers. For the deeper version of this argument, where the model routes to verified indicators instead of computing anything itself, see Supervised RAG: grounding that holds up.
The takeaway
The architecture is not exotic. What makes it work is the discipline: retrieve the right context, ground every claim in it, cite every source, signal confidence honestly, and route the questions that deserve a calculation to a calculation. None of that is a model capability you can buy. It’s a set of design decisions you have to make on purpose — which is exactly why most RAG systems that demo well still fail in production, and why the ones built with this discipline hold up.