The hardest part of running RAG in production is working out what is broken when quality dips. Was the retrieval bad? Was the answer hallucinated despite good context? Was the question ambiguous or out of scope? You cannot debug what you cannot separate. Here is the evaluation layout to put in place before a RAG system goes live, and how to keep it useful afterwards.
Separate retrieval and generation metrics
Retrieval gets its own eval set: query → expected document or chunk IDs. Generation gets a different one: (query + correct context) → expected answer. Mixing them is the most common evaluation mistake in RAG projects. When a blended metric drops, you have no idea where to look. Split the metrics and a quality dip points you at the retriever, the generator or the data.
| Retrieval | Generation (given correct context) | Where to look |
|---|---|---|
| Good | Good | Elsewhere: shifts in what users ask, prompt assembly, context truncation |
| Poor | Good | Embeddings, chunking, filters, hybrid search, reranking |
| Good | Poor | Prompt, model choice, context ordering, citation instructions |
| Poor | Poor | Start with the data: coverage gaps, stale or duplicated documents |
Retrieval metrics that actually matter
- Recall@k: does the right document appear in the top k? Pick k based on how many chunks you actually pass to the generator, not on what looks good in a report.
- MRR (mean reciprocal rank): how high does the first relevant document rank? It penalises borderline-relevant results crowding the top.
- Coverage: across all queries, how often does the system retrieve any relevant document? A low figure here usually points to an embedding model mismatch rather than chunking.
If you use a reranker, measure before and after it so you know what each stage contributes.
Generation metrics that actually matter
- Faithfulness: is the answer supported by the provided context? LLM-as-judge works here if you calibrate it against human labels from your domain.
- Answer relevance: does the answer address the question? This differs from faithfulness; an answer can be faithful and irrelevant.
- Citation accuracy: do the cited sources actually support the claims? Critical in regulated domains.
- Correct refusal: when the context does not contain the answer, does the system say so rather than guess?
Libraries such as Ragas implement many of these metrics. Treat their scores as signals to calibrate, not as ground truth.
The eval set you build, not the one you download
Public RAG benchmarks tell you whether your stack is plausible. Your own domain eval set tells you whether it works. Build 100–300 hand-curated queries with expected answers and expected sources before shipping. Teams that skip this usually end up building it in a hurry after a quality incident, which is the worst time to do it. Include:
- Real user questions from search logs, support tickets or pilot users, not only ones your team invented.
- A spread across document types, topics and user segments.
- Hard cases: multi-hop questions, questions whose answer changed over time, and questions with no answer in the corpus.
- Expected sources for every question, so retrieval can be scored without a judge.
Re-curate quarterly as your domain and documents change, and add every production failure you find as a new case.
Online metrics complete the picture
Offline evals validate against frozen expectations. Online metrics tell you how the system behaves in the wild: citation click-through, thumbs-up rate, follow-up question rate and escalation-to-human rate. Wire both. Offline regressions catch most issues before deploy; online metrics catch the rest, including changes in what users ask.
Continuous regression suite
Treat the eval set like a test suite. Run it on every change to the prompt, the retriever, the chunking or the model, and block deploys on regressions above an agreed threshold. Track metrics per category, because the average can hide a 20% drop in one user segment.
Common mistakes
- Using the same model as generator and judge without checking the judge against human labels.
- Evaluating against documents that have since been updated or deleted.
- Reporting a single headline score to stakeholders with no breakdown.
- Changing the eval set and the system at the same time, so you cannot tell which caused the movement.
FAQ
How often should we run the full eval suite?
On every change that could affect retrieval or generation, and on a regular schedule, for example weekly, to catch drift from document updates even when no code has changed.
Can we generate the eval set with an LLM?
You can use a model to draft candidate questions from your documents, but have domain experts review and edit them, and add real user questions. A set written only by a model tends to mirror that model's blind spots.
What score is good enough to launch?
There is no universal number. Agree a threshold per metric with the business owner, based on the cost of a wrong answer in your domain, and hold every category to it.
How we approach this at Velura Labs
Our RAG and knowledge systems engagements ship with retrieval and generation eval sets as deliverables, not afterthoughts, alongside custom LLM applications for the wider system and agentic systems when retrieval is one step in an agent workflow. Read our production evaluation playbook for the broader framing and our vector database decision guide for the infrastructure side. Talk to us when your RAG quality is "fine on most queries" and you cannot tell which ones.
We ship work like this for clients in the US (California, Texas, Washington, New York), across Europe (France, Italy and the EU), the Gulf (UAE and Saudi Arabia) and India — with an India delivery base that keeps cost down and time-zone overlap high. Talk to us.