This chapter emphasizes why rigorous evaluation is essential as RAG applications grow more complex. It frames evaluation around the full pipeline—from the user query through tool selection, retrieval, and answer generation—so that weaknesses can be isolated and addressed. Because different retrieval tools serve different needs (for example, vector search for semantic matching, Cypher templates for precise structured queries, and text2cypher for flexible graph querying), measuring how well the system selects and uses the right tool is critical to maintaining accuracy, coherence, and scope control.
The chapter outlines how to design a durable benchmark dataset that exercises real-world usage and edge cases. It recommends including questions that test tool selection, entity and value mapping, multi-step retrieval, ambiguous or long-tail scenarios, and conversational usability. Ground truth answers are defined with Cypher statements rather than static strings, ensuring the benchmark remains valid even as underlying graph data changes. Example categories span greetings and scope reminders, movie–actor lookups, aggregated and filtered queries that require text2cypher, and cases where data is intentionally missing, with coverage tailored to the agent’s specific capabilities.
For measurement, the chapter uses the RAGAS framework with three core metrics: context recall (whether the right information was retrieved), faithfulness (whether the answer stays grounded in the retrieved context), and answer correctness (how well the answer matches the ground truth). The workflow loads the dataset, executes Cypher to obtain ground truth, generates answers and records latency and retrieved contexts, then computes metrics. A representative run reports answer correctness at 0.7774, context recall at 0.7941, and faithfulness at 0.9657—indicating strong grounding but room to improve retrieval coverage and final accuracy. Observations include higher latency when text2cypher is invoked and a failure case on a “longest name” query, suggesting targeted fixes such as better entity mapping, query reformulation, few-shot guidance for text2cypher, or dedicated tools. The chapter concludes that systematic benchmarking enables iterative improvements that steadily raise robustness and real-world performance.
Evaluating different steps of a RAG pipeline.
Benchmark test example
Benchmark test example with a Cypher statement as ground truth
Summary
Evaluating a RAG pipeline is crucial for ensuring accurate and coherent answers. A benchmark evaluation helps measure performance and define the agent’s capabilities.
The evaluation process involves assessing various stages: retrieval tool selection, context retrieval relevance, answer generation quality, and overall system effectiveness.
A well-structured benchmark dataset should include diverse queries that test retrieval accuracy, entity mapping, handling of greetings, irrelevant queries, and various Cypher-based database lookups.
Instead of static expected answers, using Cypher queries as ground truth ensures the benchmark remains valid even if the underlying data changes.
Context recall measures how well the system retrieves relevant information.
Faithfulness evaluates if the generated answer is factually consistent with the retrieved content.
Answer correctness assesses whether the response fully and accurately addresses the query.
FAQ
Why should I benchmark a RAG application?Benchmarking ensures your agent remains accurate and coherent as the pipeline becomes more complex. It provides a structured way to measure capabilities, define scope, and isolate weaknesses across tool selection, retrieval, and answer generation.Which stages of the RAG pipeline need evaluation?Evaluate: (1) tool selection (does the LLM pick the right retrieval method?), (2) retrieval relevance (is the fetched context accurate and sufficient?), (3) answer generation (is the response coherent, correct, and grounded in context?), and (4) the pipeline holistically to spot failures and iterate.How do retrieval tools like vector search, Cypher templates, and text2cypher impact evaluation?Each tool serves different needs: vector search retrieves semantically relevant text, Cypher templates enable precise structured queries, and text2cypher lets the LLM compose flexible graph queries. Evaluating tool selection checks whether the agent picks the best method for each query.How do I design a benchmark dataset for a RAG agent?Include diverse queries that test: tool selection, entity/value mapping, multi-step retrieval, edge cases and full functional coverage, and conversational usability (greetings, clarifications, out-of-scope handling). The chapter uses a Movies dataset and shows how to cover all these categories.Why use Cypher queries as dynamic ground truth?Using Cypher to compute ground truth keeps tests valid as data changes. Instead of static strings, the database executes the truth at evaluation time, ensuring accuracy and minimizing maintenance.What is Context Recall and how is it measured?Context recall checks whether the retrieved context contains the information needed to support each sentence of the answer. A prompt classifies each sentence as attributable (1) or not (0) to the context; higher scores mean better retrieval coverage.What is Faithfulness and how is it measured?Faithfulness tests if the answer’s claims are directly supported by retrieved documents. The answer is first broken into atomic statements, then each is judged as inferable (1) or not (0) from the context, reducing hallucinations.What is Answer Correctness and how is it measured?Answer correctness compares the answer to ground truth using the same statement breakdown. Each statement is labeled as TP (supported), FP (unsupported), or FN (missing vs. ground truth), with reasons for the classification.How do I run the evaluation with RAGAS?Load the CSV benchmark, execute each test’s Cypher to get ground truth, generate answers with the agent, capture contexts and latency, then run RAGAS with metrics: answer_correctness, context_recall, and faithfulness. Fill missing answers with “I don’t know” to satisfy non-null requirements.What did the example results show, and how can I improve?The sample results were: answer_correctness 0.7774, context_recall 0.7941, faithfulness 0.9657. Retrieval coverage and answer formulation need improvement, while grounding is strong. Observations: text2cypher adds latency; LLM-as-judge can yield inconsistent scores; a failure case (“Who has the longest name among all actors?”) suggests adding few-shot guidance or a dedicated tool. Expand and refine the benchmark as the agent’s capabilities evolve.
pro $24.99 per month
access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
Essential GraphRAG ebook for free
