LLM Evaluation in Production

Most LLM demos are graded on a friendly set: ten to fifty questions someone on the team already knows the answer to, often written while staring at the same documents the retriever will see. Scores look strong. Stakeholders nod. Then real traffic arrives — ambiguous phrasing, partial product names, questions whose answer is not in the corpus, and users who ask the same thing three different ways after the first answer disappoints them.

That is why LLM evaluation in production is a different discipline from scoring a pilot. Pilot metrics measure whether the system can look competent on curated inputs. Production metrics measure whether it stays useful, grounded, and economical under the distribution you actually get. If your only number is "accuracy on the golden set," you will ship regressions that the golden set never saw.

Demo sets also hide operational failure modes: timeouts under peak load, ACL filters that empty the context window for half your tenants, and cost spikes when a prompt change doubles tokens per answer. None of those show up in a spreadsheet of hand-picked Q&A pairs. We treat evaluation as part of the same production stack described in our RAG and LLM production guide: architecture without measurement is just a diagram, and measurement without a release gate is just a report.

Offline evaluation is the labelled set you can re-run on every change. Build it from real tickets, search logs, and support transcripts — not from the prompt writer's imagination. For each item, record the question, the acceptable answer or answer criteria, and, for RAG, the passages that should be retrieved or cited. Score retrieval and generation separately so you know which half of the stack broke.

Online signals are what happen after deploy: thumbs-down rates, escalation to a human, empty-result rates, citation click-through, time-to-resolution, and the share of answers that trigger your refusal path. Online data is noisy and delayed, but it is the only view of questions your offline set has not imagined yet. Use offline to decide whether a change may ship; use online to decide whether it earned the right to stay.

Teams that only watch online metrics discover failures after users do. Teams that only watch offline metrics optimise for a museum of yesterday's questions. Production LLM testing needs both, with clear ownership of each.

A workable split looks like this. Product and domain experts approve what belongs in the offline set and which failures are unacceptable. Engineering owns the harness, the CI job, and the dashboards. Risk or compliance reviews thresholds for high-stakes intents. When those roles blur, the suite either freezes (nobody dares change a labelled answer) or drifts (engineers rewrite expected outputs until the scoreboard is green again). Write the ownership down next to the suite version.

Refresh the offline set on a cadence. Every week, sample failed or escalated online queries, strip sensitive fields, and add the ones that represent a new failure mode. Retire items that no longer match the product. A stale golden set is how teams pass the gate while users quietly leave.

Fluency is cheap. A model can produce a confident paragraph that sounds like your policy manual and quietly contradicts the passages you retrieved — or invents a clause that was never there. For enterprise RAG, faithfulness (is the answer supported by the retrieved context?) matters more than stylistic polish.

Practical checks we use on real systems:

Score answers against the retrieved passages, not against the entire corpus. If the right document was never retrieved, that is a retrieval failure; do not punish the generator for it, and do not celebrate a fluent answer that ignored weak context.

Require citations users can open. An answer without a source is an opinion. In regulated workflows, an unsourced answer is often worse than a clean refusal.

Measure refusal quality. "I could not find this in the approved sources" on an out-of-corpus question is a success. Inventing a plausible answer is a failure, even if a generic helpfulness rubric would reward it.

These LLM eval metrics sit on top of the retrieval fundamentals in building RAG systems that survive production. If recall is weak, faithfulness scores will lie to you — the model is faithful to the wrong passages.

Be careful with automated judges. An LLM-as-judge can triage at scale, but it inherits biases from its own prompt and can reward fluent nonsense that matches a vague rubric. Use judges to rank or flag; keep a human-labelled core set as the release authority for anything that can hurt a customer, a patient, or a regulated process. Report judge agreement against that core set so you know when the judge itself drifted.

For comparative work — model A versus model B, or prompt v3 versus v4 — run paired evaluation on the same items and look at win/loss by intent category, not a single headline score. A change that wins on "summarise this policy" and loses on "what is the retention period for X" is not a wash if retention questions are the ones that create audit risk.

Quality metrics that ignore latency and cost produce systems nobody can afford to run. Track p50 and p95 end-to-end latency, tokens per successful answer, and cost per resolved task — not cost per raw API call. A change that lifts faithfulness by two points and doubles token spend may be a net loss once you include the human time still spent checking answers.

When quality and cost move in opposite directions, make the trade-off explicit. Sometimes a smaller model with tighter retrieval wins on the joint objective. Sometimes you reserve the large model for high-risk intents and route the rest elsewhere. Evaluation should surface that choice, not hide it behind a single "accuracy" number.

Instrument the stages, not only the end-to-end call. Retrieval time, rerank time, generation time, and tool calls tell you where a regression landed. A faithfulness dip paired with a sudden jump in context tokens usually means someone widened top-K or chunk size to "help" quality and accidentally bought noise. Put those stage metrics next to the LLM eval metrics so the fix is obvious.

Budget alerts should fire before finance does. Cap cost per intent class, and treat a silent doubling of tokens on a high-volume path the same way you would treat an SLO breach — investigate, then gate the change that caused it.

A production LLM stack changes constantly: prompt tweaks, chunking experiments, embedding upgrades, model swaps, reranker thresholds. Without a gate, every change is a vibe check. Define a minimum offline suite that must pass before promote:

A fixed labelled set with versioned expected citations where RAG is in play. A faithfulness or groundedness threshold on that set. A latency budget on a representative hardware or API path. A cost ceiling per successful answer for the top intents. A smoke path for refusal and "not in corpus" behaviour.

Store the last known-good config and scores. If the suite fails, the change does not ship — or it ships only behind a flag to a small cohort with an explicit rollback. This is ordinary release engineering applied to language systems. It is also the difference between "we evaluate LLMs" as a slide and LLM evaluation as an operating practice.

Version everything the gate depends on: prompt templates, tool schemas, chunking config, embedding model ID, reranker ID, generator model ID, and the suite itself. When a score moves, you need to know which knob turned. Without that, teams argue about feelings while production absorbs the risk.

Shadow mode helps for large model swaps: serve the candidate in parallel, score it offline against production traffic samples, and only cut over when the gate and a soak period agree. Do not confuse a successful shadow with a production soak — permissions, caching, and peak-load behaviour still need the live path.

Evaluation does not end at launch. Ops needs a short list of alerts that mean "wake someone," not a wall of model-internals graphs. Useful wakes: sudden drop in citation validity, spike in refusals or escalations for a core intent, latency past SLO, cost per task above the agreed envelope, or a retrieval empty-hit rate that suggests index or ACL breakage.

Pair that with a cadence: weekly review of failed online samples into the offline set, monthly revisit of thresholds with the business owner, and a named person who can roll back the prompt or model without a committee. The same discipline that keeps classical ML alive — monitoring, drift awareness, rollback — applies here, even when the "model" is a hosted API.

If you are sequencing this work against a wider production roadmap, start from the pillar guide and the retrieval essay linked above. When you want a partner to stand up the harness and the gates — not another slide deck of benchmark names — talk to us about enterprise AI consulting or generative AI and LLM development, or contact Rowth.ai with how your system is scored today.

Security and access control change what "correct" even means in multi-tenant corpora; that thread continues in RAG security and governance for enterprises.