2 Nothing happens until someone writes an eval

This chapter makes the case that effective RAG projects begin with evals—concrete question–answer test cases that define success before any code is written. Adopting an eval‑driven development mindset (akin to test‑driven development) turns vague objectives into measurable targets, aligns stakeholders on expectations, and provides a practical roadmap for progress. Evals clarify scope, set pass/fail criteria, and help teams focus on what matters for the business, laying the foundation for a reliable, purpose‑built RAG chatbot.

The workflow starts by collecting 10–20 representative questions and correct answers from stakeholders, subject‑matter experts, and real customer interactions. You iteratively build the system to pass these evals, diagnosing misses as either retrieval failures or generation mistakes, then refining prompts, data coverage, or logic until answers match expectations. Evals serve as guardrails against scope creep, a communication tool when new requests arise, and an end‑to‑end diagnostic aid after the system is live. To ensure comprehensive coverage, include evals for every data source your chatbot will query and re‑run earlier tests whenever you improve later ones to avoid regressions.

To scale quality assurance, the chapter shows how to automate evals in a CI/CD pipeline. A test harness sends eval questions to the chatbot and compares responses to expected answers, often using an LLM as a semantic judge. Because LLM judgments can be variable and costly, combine them with rule‑based checks (for exact numbers, units, or required terms) and occasional human oversight. Automated evals catch regressions early, enforce consistency and safety requirements, and provide continuous feedback as prompts and code evolve. Practically, you wire this up with lightweight scripts, environment‑secured keys, pytest, and GitHub Actions—starting with failing tests and iterating until they pass.

The iterative process for building a RAG system using evals. Steps include defining evals, implementing the system, testing, modifying prompts, and retesting for refinement.

Using evals as test cases in CI/CD pipelines. Evals ensure the chatbot consistently meets performance standards by running, modifying, and retesting prompts in a continuous feedback loop.

How the Automated Eval process works. Evals interact with the chatbot and large language model (LLM) to assess and ensure response accuracy, passing or failing based on similarity.

The automated eval system for RAG development. Evals compare chatbot answers to expected outputs and automate testing via GitHub Actions, ensuring high-quality performance.

• Evals are essential for RAG Development. Evals serve as test cases that transform broad objectives into specific, measurable goals, guiding the development of your Retrieval Augmented Generation (RAG) chatbot.
• Obtain Evals Before Coding. Getting evals from stakeholders before writing any code is crucial to avoid hidden assumptions, ensure alignment with expectations, and protect the project from misaligned outcomes.
• Utilize a step-by-step approach by starting with simple question-answer pairs, identifying required data, building agents and search functions, providing clear instructions, testing responses, troubleshooting issues, and iterating.
• Integrate evals into your deployment pipeline to automate testing, use Large Language Models (LLMs) for response comparison, and cover all data sources to ensure comprehensive and efficient evaluation.
• Evals enhance communication between developers and stakeholders, enable continuous improvement, and are critical for developing, testing, and maintaining a reliable RAG chatbot that meets organizational needs and adds substantial value.

FAQ