Production LLM Evaluation: Catch Regressions Before Users Do

Most LLM projects ship without evals. The team eyeballs ten outputs, decides the demo looks good, and pushes. Then a model upgrade lands, output quality silently degrades, and the first signal is a customer complaint two weeks later. Evals are the cheapest insurance against this story, and teams often skip them because they don't know where to start. Here is where to start.

Three layers of evals you actually need

• Unit-level prompt evals. For each prompt template, a small set of golden inputs with expected output properties. Run on every PR.
• End-to-end task evals. Realistic user queries scored by a frontier-model judge (LLM-as-judge) plus rule-based checks. Run nightly and on every model upgrade.
• Production observability. Sample real traffic, log it to Langfuse or LangSmith, score it asynchronously, and surface drift on a dashboard.

The layers catch different things. Unit evals catch a broken prompt edit in minutes. Task evals catch quality shifts across the whole flow, including retrieval. Production sampling catches the inputs nobody thought to test.

The metrics that survive contact with reality

RAGAS gives you faithfulness, answer relevancy and context precision out of the box. That is a useful baseline for retrieval systems. For production, though, three metrics matter more than the rest:

• Task-specific exact match or schema validity. If you are extracting JSON, did the JSON parse? Did every required field show up?
• Refusal correctness. When the model should say "I don't know", does it? Hallucination is one of the most damaging production failure modes, and it is testable: include questions your sources cannot answer and check that the system declines.
• Latency budget. Track p50 and p95. A model that is right but too slow for the feature can be worse than one that is slightly less accurate and fast. Set the budget per feature.

Golden datasets: small, hand-curated, ruthlessly maintained

Your golden set should have 50 to 200 examples, hand-picked to cover the failure modes that matter. Update it whenever a new failure mode shows up in production; that is the whole loop. Auto-generated synthetic evals are a starting point but never a substitute. A practical way to build one:

1. Pull real inputs from logs or support tickets, with personal data removed.
2. Tag each with the failure mode it covers: ambiguous question, missing context, out of scope, adversarial, long input, non-English.
3. Write expected properties rather than exact outputs: must cite a source, must refuse, must return valid JSON with the required fields.
4. Version the set in the repository alongside the prompts.
5. Add at least one example for every production incident.

Making LLM-as-judge trustworthy

A judge model is another model, and it can be wrong. Keep it honest:

• Use a rubric with a few discrete scores (pass/fail, or 1 to 3) rather than a 1 to 10 scale.
• Ask for the reasoning before the score.
• Calibrate: have a person label around 50 items and check how often the judge agrees before you trust it.
• Pin the judge model version, so a judge upgrade does not show up as a product regression.
• Prefer rule-based checks wherever they apply. They are cheaper, faster and deterministic.

Regression budgets and the CI gate

A gate that blocks on any score drop gets ignored within a week, because LLM outputs are noisy. Set a budget per metric instead. For example: schema validity must stay at 100%, faithfulness may not fall more than two points below the current baseline, and p95 latency may not rise more than 15%. When a result sits close to a threshold, rerun the set before deciding. When a change is deliberately worse on one metric and better on another, record the decision and reset the baseline.

The cheapest way to wire this up

Braintrust, Promptfoo, OpenAI Evals or a custom JSON Lines runner will all work. We default to Braintrust for client work because the dashboard reads well in client meetings, and Promptfoo for open-source builds. Whatever you use, run it in CI as a gate before merge to main.

Treat model upgrades as releases

Every provider model change is a release, even when your code has not changed. Run the full end-to-end set against the new version side by side with the current one, compare results per failure-mode tag, and read a sample of the differences yourself. Aggregate scores can hide a new failure in a small but important category.

Common mistakes

• Judging quality by reading a handful of outputs.
• Relying only on synthetic data that never resembles real user input.
• Trusting an LLM judge that nobody has calibrated.
• Building an eval suite that only runs when someone remembers to run it.
• Never sampling production traffic, so the golden set goes stale.

FAQ

How long does a first eval harness take to build?

A basic version (a golden set of about 50 examples, a runner and a CI step) is usually a matter of days rather than weeks. The ongoing work is keeping the set current.

Do we need evals for a simple chatbot?

Yes, if customers see its answers. Even a small set of refusal and accuracy checks will catch the worst regressions after a model or prompt change.

How we apply this at Velura Labs

Every Custom LLM Application we ship comes with an eval harness wired into CI. For agent-heavy systems, the harness extends into Agentic Systems: golden trajectories, replay on every PR and regression budgets. If your team has shipped LLMs without evals and the dashboards are starting to look noisy, multilingual RAG isn't your problem yet; eval discipline is. Get in touch and we'll review your eval coverage with you.

Available to businesses across the United States (Washington, California, Texas, New York), Europe (France, Italy and the wider EU), the Middle East (Dubai and the Gulf) and India. Get in touch to scope your build.