The gap between a RAG demo and a RAG product
A retrieval-augmented generation demo is one of the most seductive things in software. Point a language model at a folder of documents, add a vector search, and within an afternoon you have something that answers questions about your own data. The problem is that the demo and the product live in different universes. The demo is judged on whether it produces a plausible answer to a friendly question. The product is judged on whether it produces a correct, cited answer to a hostile question, every time, for years, without leaking anything it should not. Closing that gap is the entire job.
Retrieval quality is the whole ballgame
Most teams obsess over the model and neglect retrieval, which is backwards. If the right passage never makes it into the context window, no amount of model quality will save the answer. Retrieval quality starts with how you split documents: chunk too coarsely and you bury the relevant sentence in noise; chunk too finely and you sever the context that makes a passage meaningful. Embeddings matter, but so do the unglamorous details, such as preserving document structure, attaching metadata for filtering, and handling tables and figures rather than flattening them into gibberish. When retrieval is the bottleneck, improving it returns more than swapping models ever will.
Rerank, because first-pass retrieval is noisy
Vector similarity is a fast, blunt instrument. It will happily return passages that are topically close but answer a different question. A reranking step, ideally a model trained on your own labelled pairs, reorders the candidates so the genuinely relevant passages rise to the top before they reach the language model. In high-stakes domains a reranker trained on expert-labelled pairs is frequently the highest-leverage component in the whole pipeline, because where the gap between the second-best and the best passage is the gap between a useful answer and a dangerous one, ordering matters more than raw recall.
Citations and provenance are non-negotiable
An answer without a source is an opinion, and in regulated settings an unsourced answer is worse than no answer at all. Every response a production RAG system produces should carry provenance: which documents it drew from, and ideally which passages. This does two things. It lets the user verify the claim instead of trusting the machine, and it gives you an audit trail when someone asks, months later, why the system said what it said. Attaching a citation to every answer, and putting the cited passage one click away, is usually what turns a clever tool into one a regulated function will actually sign off.
You cannot improve what you do not evaluate
Teams routinely ship RAG with no evaluation harness, then tune it by vibes. Build a real evaluation set drawn from the questions your users actually ask, with known-good answers and known-good sources. Measure retrieval recall separately from answer quality, so you know which half of the system to fix. Track faithfulness, the degree to which the answer is supported by the retrieved passages, because a fluent answer that quietly contradicts its own sources is the most dangerous failure mode in the whole category. An evaluation harness turns RAG from an art into an engineering discipline.
Guardrails, tenancy, and the operational details
Production RAG lives inside real constraints. If you serve multiple business units or customers, retrieval has to respect tenancy so one tenant can never surface another's documents. You need guardrails for prompt injection, for personally identifiable information, and for the model's tendency to answer confidently when it should decline. None of this is glamorous, and all of it is what separates a system that survives its first security review from one that does not. The same discipline applies to any enterprise RAG deployment, whether the corpus is clinical protocols, legal contracts or the internal knowledge base nobody has audited in three years.
The takeaway
Retrieval-augmented generation rewards the teams who treat retrieval, evaluation, and provenance as first-class engineering problems rather than afterthoughts to a model. Get those right and RAG becomes a genuinely transformative interface to your organisation's knowledge. Get them wrong and you ship a confident liar. Our generative AI and LLM practice builds RAG systems for the second universe, the one where the answers have to be right.