Chapter 10 explains that evaluating AI agents is essential because their autonomy can amplify small mistakes into real-world harm, from incorrect actions to costly side effects. Unlike simpler AI systems, agents must be judged not just on final answers but on the reasoning, tool choices, and action sequences that produced them. Since those behaviors are complex and highly variable, evaluation needs to be automated to keep development from slowing to a crawl.
The chapter then shows how effective evaluation depends on observability: collecting telemetry such as metrics, traces, and logs so the agent’s internal behavior can be inspected after the fact. It outlines a practical process for building evaluation datasets from benchmarks, custom scenarios, and production data, then analyzing failures through open coding and a failure taxonomy. From that analysis, teams derive rubrics and trackable metrics so quality can be measured consistently rather than guessed at.
Finally, the chapter presents multiple ways to operationalize evaluation. LLMs can act as judges for scalable scoring, while code-based evaluators, human reviewers, and end-user feedback each cover different strengths and weaknesses. The chapter closes by tying evaluation into CI/CD, where automated gates prevent weak changes from reaching production, and by emphasizing a continuous improvement loop in which real usage data repeatedly updates datasets, rubrics, and metrics as the agent evolves.
An overview of chapter 10.
Workflow for defining agent evaluation criteria: From open-coding to defining metrics.
Three aspects to consider in agent evaluation.
Agent Quality Flywheel: A continuous four-step loop for improving agent quality. The process begins by defining quality criteria (Step 1), followed by instrumenting the system to gain visibility into real-world behavior (Step 2). The collected data is then used to evaluate the agent’s performance and identify failure patterns (Step 3). Finally, feedback loops are architected to update datasets, rubrics, and metrics (Step 4), feeding insights back into Step 1 to refine how quality is defined.
Summary
As agent autonomy increases, failures shift from minor bugs to real-world impacts with meaningful cost.
Evaluation must cover not only the final output, but also the components and trajectory to separate luck from reproducible success.
Instrumentation/observability is required because evaluation is impossible without execution records.
Strong evaluation starts by converting failures into a shared language via trace review, open-coded error logging, and a failure taxonomy.
Convert the most frequent and severe failures into a PASS/FAIL-first rubric; metrics are a dashboard that turns rubrics into numbers to catch regressions and guide improvement.
Embed automated evaluation into the deployment pipeline.
Treat evaluation as a continuous flywheel because environments, users, and tools/APIs change: operational data → error analysis → taxonomy → rubric/metrics → test set updates.
FAQ
Why is automated evaluation essential for AI agents?Agent evaluation must be automated because agents are non-deterministic, multi-step systems with many possible action paths and failure points. Checking only the final output can miss whether the result was achieved through sound reasoning or by luck. Automation is necessary to evaluate outputs, intermediate reasoning, tool use, and trajectories at scale, without making evaluation a development bottleneck.What parts of an agent should be evaluated?Agent evaluation should cover three main perspectives: the final output (did it meet the request), the components (did tools, retrieval, code execution, and other modules work correctly), and the trajectory (was the reasoning and tool-call path reasonable and efficient).What is observability in the context of agents?Observability means making the internal operation of an agent visible, including input handling, reasoning, tool usage, and final output. It helps developers debug issues, preserve logs for evaluation, and understand non-deterministic LLM behavior over time.What telemetry signals are most useful for agent evaluation?Useful telemetry includes metrics, traces, and logs. Examples include latency, token usage, API call counts, tool-call failures, reasoning traces, and error reports. These signals form the raw data layer for later analysis and evaluation.How are metrics, traces, and logs different?Metrics are numerical measurements that show that something is happening, such as latency or error rate. Traces show the full path a request took through the system, including spans and events. Logs provide detailed contextual records of what happened, such as errors or tool inputs and outputs.Why is OpenTelemetry used for agent observability?OpenTelemetry is a vendor-neutral standard for generating, collecting, and exporting telemetry data. It helps teams instrument LLM calls, tool invocations, planning steps, and other key events without lock-in to a proprietary observability platform.How do you build an evaluation dataset for an agent?Evaluation datasets are usually built from three sources: benchmarks (public datasets like HotpotQA, GAIA, or SWE-bench), custom datasets (internal scenarios reflecting product use cases), and production data (real user interactions collected after deployment). A strong test set usually mixes core cases, edge cases, and adversarial inputs.What is error analysis and why is it important?Error analysis means reviewing execution traces to identify recurring failure patterns before defining metrics. Evaluators write open-ended notes about what went wrong, group similar issues into a failure taxonomy, and use those findings to design rubrics and metrics grounded in real behavior.What is LLM-as-a-Judge?LLM-as-a-Judge is an evaluation method where a large language model grades another system’s outputs using a rubric. It is useful because it scales much better than human evaluation and can handle many natural-language tasks where exact code-based metrics are not enough.How does evaluation fit into CI/CD for agents?Evaluation should be integrated into CI/CD so agent quality is checked continuously. In CI, small fast test sets catch regressions early; in staging, heavier validation tests more realistic conditions. Deployment can be gated so unverified changes do not reach production.
pro $24.99 per month
access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
Build an AI Agent (From Scratch) ebook for free
