14 Evaluating RAG systems
Evaluating a RAG system is not the same as evaluating ordinary retrieval, because good search scores can hide bad answers. Classical metrics like precision, recall, MRR, and NDCG are still useful, but they mainly measure whether retrieved documents are topically relevant to the query, not whether the language model can actually use them well. In RAG, rank position matters, irrelevant material can crowd out useful context, and a document that looks relevant on paper may still fail to support a correct response. The chapter argues that the right unit of evaluation is the whole pipeline, not just retrieval in isolation.
To measure the full pipeline, the chapter presents a RAG-specific triad: faithfulness, answer relevance, and context relevance. Faithfulness checks whether the response is grounded in the supplied context, answer relevance checks whether the response actually answers the question at the right level of specificity, and context relevance checks how much of the retrieved context is genuinely useful. These scores are intentionally diagnostic rather than absolute: they are best used comparatively, to spot regressions and identify whether retrieval or generation is the main source of failure. The chapter also surveys LLM-as-judge methods such as G-Eval and ARES, noting both their usefulness and their biases, including self-preference, prompt sensitivity, and variance across runs.
The chapter then shows how to build evaluation data and pipelines that make these metrics practical. Because hand-labeled benchmarks are often skewed toward easy factual questions, it recommends generating synthetic datasets from your own corpus with a balance of factual, summary, reasoning, and unanswerable questions. With those datasets in place, an automated pipeline can score each query-context-response triple, map low scores to specific failure points, and guide fixes such as reranking, better chunking, grounding prompts, or query reformulation. The closing case study shows this workflow in action: a compliance assistant improved through successive evaluation cycles, first fixing retrieval, then addressing faithfulness regressions, and finally exposing a hidden weakness on reasoning questions that only a balanced test set revealed.
Precision and recall. Of the items returned by the retriever (the selected set), precision measures the fraction that are relevant. Of all relevant items in the corpus, recall measures the fraction that the retriever surfaced.
Traditional vs LLM-based retrieval evaluation.
The RAGAS evaluation triad.
The ARES three-stage evaluation pipeline.
The complete evaluation pipeline.
Summary
- Retrieval metrics alone mislead: Classical metrics like Precision@k and NDCG evaluate relevance to the query, not utility to the generator. eRAG showed that per-document LLM-based evaluation achieves substantially higher correlation with actual RAG performance (Kendall's tau improvements of 0.168-0.494).
- The RAGAS triad decomposes quality into three separable dimensions: Faithfulness (is the answer grounded in context?), answer relevance (does it address the question?), and context relevance (is the retrieved context focused?). Each dimension targets specific failure points from Table 1.1, enabling precise diagnosis.
- LLMs can evaluate without ground truth: Reference-free evaluation (using LLMs to judge faithfulness via claim extraction, relevance via reverse question generation, and context quality via sentence extraction) makes evaluation feasible for enterprise domains where reference answers don't exist.
- G-Eval improved LLM judging through chain-of-thought and probability-weighted scoring: Generating evaluation steps before scoring increases correlation with human judgments. ARES refined this further with fine-tuned lightweight judges and prediction-powered inference, providing confidence intervals that distinguish real regressions from noise.
- Public benchmarks are skewed: Datasets like NaturalQuestions and SQuAD are dominated by fact_single questions (60-80%), underrepresenting the summary and reasoning queries common in enterprise use. Generating synthetic evaluation data from your own corpus using statement extraction produces balanced, domain-representative test sets.
- Evaluation maps to action: The diagnostic mapper connects RAGAS scores to specific failure points (FP1-FP7) and recommended techniques from earlier chapters, turning vague quality complaints into targeted engineering work.
- A score without an interval cannot be compared: Automated judges agree with human assessors far more closely on which system is better than on how good any single answer is, so read every number against another one and compute the interval before acting on a difference. Group the resampling by whatever the questions were generated from, because questions built from one passage are not independent observations.
- Evaluation is continuous, not one-time: A production RAG system needs an evaluation flywheel: generate test data, score, diagnose, fix, remeasure. Different improvements can interact: better retrieval can expose generation weaknesses that were hidden by noisy context, and only continuous measurement catches these effects.
Retrieval Augmented Generation, The Foundational Ideas ebook for free
