RAG & LLM Systems in Production

A retrieval-augmented generation demo is easy to love. Point a language model at a folder of documents, add vector search, and within an afternoon you have something that answers questions about your own data. That demo and a production system live in different universes. The demo is judged on whether it produces a plausible answer to a friendly question. Production is judged on whether it produces a correct, cited answer to a hostile or ambiguous question — every time, for years, without leaking another tenant's documents or inventing a policy that never existed.

Three failure modes show up first. Retrieval returns a passage that is topically close but answers a different question, so the model answers confidently from the wrong context. There is no evaluation harness, so every prompt edit is a vibe check and nobody can tell whether last week's change helped or quietly regressed a category of question. Cost climbs as teams stuff more context into the prompt to paper over retrieval problems, and token spend scales faster than usage. None of these is fixed by swapping to a larger model.

There is a fourth failure that looks like success until audit day: answers without provenance. In regulated or customer-facing settings an unsourced answer is often worse than no answer. Production RAG has to treat citations, tenancy, and decline-to-answer behaviour as product requirements, not polish. This guide is that production view — architecture, retrieval quality, evaluation, security, and the MLOps habits that keep a RAG or LLM system honest after launch. For a deeper dive on the retrieval layer alone, see our field notes on building RAG systems that survive production.

Enterprise RAG is a pipeline, not a single API call. A workable reference shape looks like this:

1. Ingestion: clean sources, version documents, strip duplicates and obsolete copies, attach metadata (owner, classification, freshness, tenant).
2. Chunking and embedding: preserve document structure; choose chunk size for how questions are actually asked, not for a default in a tutorial.
3. Retrieval: hybrid keyword-plus-vector search with metadata filters, then a rerank pass before anything reaches the model.
4. Generation: a model behind a gateway, with pinned versions, rate limits, and prompt templates treated as code.
5. Citations and guardrails: every answer carries provenance; PII and prompt-injection checks sit on the boundary.
6. Evaluation and ops: labelled sets, latency and cost budgets, drift and correction-rate dashboards, rollback.

Fine-tuning belongs late, if at all. Retrieval handles knowledge that changes and must be attributable; fine-tuning handles stable form and task behaviour. Teams that fine-tune to fix factual errors usually needed better retrieval. Multi-step agents compound their own mistakes — a wrong decision at step two is reasoned on at steps three through seven — so autonomy should be scoped explicitly, with write access behind approval until eval data justifies opening it.

Platform choices — Postgres with pgvector versus a dedicated vector store, managed APIs versus self-hosted inference — should follow data residency and filter patterns, not fashion. Shared substrate (gateway, audit log, chargeback) is what stops the fourth use case from reinventing nine incompatible stacks. The architecture above is deliberately boring. Boring is what passes a security review and still works on a Monday morning.

If the right passage never enters the context window, no amount of model quality will save the answer. Chunk too coarsely and you bury the relevant sentence in noise; chunk too finely and you sever the context that makes a passage meaningful. Embeddings matter, but so do preserving headings and tables, attaching filterable metadata, and refusing to flatten figures into gibberish.

First-pass vector similarity is a fast, blunt instrument. It will happily return passages that share vocabulary with the question and answer something else. A reranker — ideally tuned on labelled pairs from your domain — reorders candidates so the genuinely useful passages rise before generation. In high-stakes domains that ordering gap is the difference between a useful answer and a dangerous one.

Hybrid search earns its keep wherever exact identifiers appear: policy names, SKUs, error codes, contract clauses. Semantic search alone underperforms on those queries; keyword or sparse retrieval beside dense vectors closes the hole. Parent-document retrieval and metadata filters (tenant, product line, effective date) are ordinary production tools, not research extras. When retrieval is the bottleneck, improving it returns more than swapping models ever will — a theme we return to in RAG systems in production.

Basic RAG assumes retrieved documents are good enough. Corrective RAG inserts an evaluation step before generation: if retrieval looks weak, rewrite the query, expand the candidate set, switch retrievers, hit another approved source, or decline to answer. That feedback loop matters when a wrong answer has real operational cost — legal, clinical, financial, customer-facing policy.

It is not free. Extra evaluation and retry paths add latency, tokens, and observability burden. A practical pattern is risk-based correction: high-confidence retrieval answers immediately; uncertain or high-risk questions take the corrective path. If forty percent of queries need correction, your ingestion or chunking is the deeper problem. If two percent do, the corrective layer is doing its job. Start simple, measure, then add correction where the data shows retrieval quality failing — not on day one of every prototype.

Teams that ship RAG without an evaluation harness tune by anecdote. Build a labelled set from questions users actually ask, with known-good answers and known-good sources. Include adversarial and out-of-corpus examples on purpose — those are the queries that expose confident nonsense. Measure retrieval separately from generation so you know which half to fix. Track faithfulness — whether the answer is supported by the retrieved passages — because a fluent answer that quietly contradicts its sources is the most dangerous failure mode in the category.

Useful production metrics include precision and recall at K, citation accuracy, answer correctness on the labelled set, faithfulness, hallucination or unsupported-claim rate, p95 latency, cost per request, and correction rate if you run Corrective RAG. Re-run the suite on every material change: chunking, embedding model, prompt, reranker, or base LLM. Without that gate, a provider-side model update or a well-meant prompt edit can regress a whole category of questions with nobody noticing until users complain.

Evaluation is also how you defend budget. Offline accuracy alone will not survive a CFO conversation; pair quality metrics with cost and latency envelopes, and with the business metric a named owner reads on Monday morning. For framing the commercial case, see measuring AI ROI.

Token spend that scales faster than engagement is usually an architecture smell, not a pricing problem. The fixes are concrete. Prune RAG context with a relevance threshold so background noise never reaches the model. Structure prompts so static system rules and reference material sit in a cacheable prefix. Split monolithic instructions into modular steps so a small model can format or validate while a larger one handles synthesis.

Most important: do not call the large model for work that is not reasoning. A hybrid router can send simple navigational or lookup intents to classifiers and traditional search, and reserve the generative LLM for tasks that need synthesis, translation, or multi-step reasoning. That pattern — detailed in reserving LLMs for reasoning — is how we cut token waste on production workloads without sacrificing answer quality where it matters. Cost problems that look like "we need a cheaper model" are often "we are retrieving too much and routing too little."

Production RAG lives inside real constraints. Multi-tenant retrieval must enforce isolation so one business unit or customer never surfaces another's documents — preferably as a hard filter at query time, not a prompt instruction the model might ignore. Guardrails cover prompt injection, PII leaving the boundary, and the model's habit of answering confidently when it should decline. Logging must be useful for audit without becoming a second store of sensitive content; retention limits on prompt and completion logs are part of the design, not an afterthought.

Sophisticated buyers ask where data is processed, who can access it, and which legal regime applies. Design for that up front: redaction before inference where the use case allows, region-pinned processing where residency is contractual, and an audit trail that can show what was asked and answered. Classification under regimes such as the EU AI Act follows the decision the system makes, not how large the model is. A modest classifier in a regulated decision path carries obligations a larger drafting model used internally may not.

Governance bolted on after a compliance scare is expensive; governance as a property of every stage — bias and risk during scoping, data handling during build, drift and human review in production — is what lets you move without waiting for the next audit scare. Document authority and freshness metadata also belong here: a current legal policy should outrank a stale wiki page even when embeddings score them similarly.

A model in production is a depreciating asset unless someone owns its health. For RAG and LLM applications that means more than uptime checks. Watch retrieval latency and empty-result rates, generation latency and token cost, faithfulness regressions on a canary labelled set, and user override or thumbs-down rates where humans sit in the loop. Pin model versions behind a gateway; adopt new versions only after the evaluation suite passes. Keep a rollback path that does not require a heroics weekend.

Ingestion pipelines need the same discipline as training pipelines: when source documents change, embeddings and indexes must update on a known schedule with lineage you can explain. Prompt templates and chunking configs belong in version control with review, not in a shared notebook. Chargeback and per-team quotas on the gateway create downward pressure on waste; without them GPU and API spend has no owner.

Incident response for LLM systems looks different from classical ML. A bad prompt deploy can look like a product regression with no infrastructure alert. Treat prompt and retrieval config changes as releases: review, staged rollout, eval gate, and a one-click revert to the last known-good bundle. Platform engineering and MLOps are unglamorous here, and they are what make the fourth use case cost a fraction of the first.

Not every team needs an outside partner. If you have a small, low-risk corpus, a working evaluation set, and a clear owner for cost and quality, keep shipping. Bring help when the pattern is stuck: pilots that cannot define success in latency, cost, or override rates; security reviews that block go-live because tenancy and audit were never designed; retrieval quality that will not move despite model swaps; or a roadmap of demos with no production path.

That is the work we do across enterprise AI consulting — sequencing use cases and production definitions that survive the CFO and the security review — and generative AI and LLM development — production RAG, evaluation harnesses, and cost engineering for systems real users depend on. If you want a concrete read on your current pipeline, talk to us.

Related in this series: Building RAG systems that survive production · What is Corrective RAG? · Reserving LLMs for reasoning. Continue with LLM evaluation in production and RAG security and governance for enterprises.