What makes retrieval-augmented AI actually work

Most RAG systems fail in the same predictable ways. This is a working architecture, the failure modes we have seen in production, and the design decisions that separate a demo from something you can put in front of a board.

The three failures we see most

Nearly every broken RAG system we are asked to review fails on one of three axes:

1. Retrieval that returns plausible-but-wrong context — the model answers confidently from the wrong chunk.
2. No grounding contract — the model is free to answer from its own priors when retrieval comes back thin.
3. No confidence signal — every answer looks equally authoritative, so users can’t tell a solid finding from a guess.

The grounding contract

The single most important design decision is to forbid the model from answering outside the retrieved context. In practice that means a system prompt that treats retrieved passages as the only source of truth:

def build_prompt(query: str, chunks: list[Chunk]) -> list[dict]:
    context = "\n\n".join(f"[{c.id}] {c.text}" for c in chunks)
    system = (
        "Answer ONLY from the numbered context below. "
        "Every claim must cite a source id like [3]. "
        "If the context does not answer the question, say so — do not guess."
    )
    return [
        {"role": "system", "content": system},
        {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {query}"},
    ]

That last instruction — say so, do not guess — is what converts a confident hallucination into an honest “the data doesn’t cover this.”

Confidence, made legible

Retrieval scores are a signal, not a verdict. We map them to labels a non-technical reader can act on:

A finding a reader can trace to its source is worth more than a summary they have to take on faith.

The architecture is not exotic. What makes it work is the discipline: ground every claim, cite every source, and never let the system sound more certain than the evidence allows.