1 The importance of AI evaluation
AI evaluation is presented as the disciplined process of measuring how an AI system behaves in the real context where it will be used, not just how well a model scores on a benchmark. The chapter opens with examples of costly failures in pricing, hiring, and customer service to show that AI systems can create serious financial, legal, and social harm after deployment. These incidents illustrate why evaluation must look beyond simple accuracy and ask whether outputs are trustworthy, useful, safe, and appropriate for the people and institutions affected.
The chapter also clarifies what counts as an AI system and why that matters for assessment. An AI system is defined by what it is intended to do, whether it predicts, generates, or takes actions, and the text emphasizes that a model is only one component of a larger product that also includes interfaces, workflows, data pipelines, and human oversight. Because users interact with the full system rather than the model alone, evaluation has to account for the whole experience and for the fact that many real-world systems are hybrid, combining predictive, generative, and agentic elements.
Finally, the chapter explains why AI evaluation has become both essential and difficult. Modern AI is more capable, more widely deployed, and more opaque than traditional software, yet current evaluation practices often remain shallow, late-stage, and overly dependent on limited metrics or unrepresentative data. To address this gap, the chapter recommends a systematic, iterative approach: define the goal and scope, choose the right metrics, gather suitable data, compute results carefully, interpret them in context, and turn findings into decisions. The overarching message is that reliable AI requires continuous evaluation supported by human judgment, not one-time testing before launch.
The difference between an AI model and an AI system. A model is a self-contained computational component that transforms inputs into outputs, while an AI system combines the model with s user interfaces, data pipelines, integration logic, human-in-the-loop mechanisms, and other functionalities and features to perform real-world tasks.
Three broad categories of AI systems: predictive systems that make forecasts or classifications, generative systems that create content, and agentic systems that take actions to achieve goals in an environment.
The six main steps of an AI evaluation project, typically executed in an iterative fashion. All six are essential, and their order matters.
Summary
- AI capabilities have surged while evaluation practices lag, and high-profile failures show the cost of that gap.
- A widening capability–comprehension gap creates two opposing risks: over-reliance and under-reliance.
- AI evaluation means systematically assessing how systems perform under real-world complexity and unpredictability.
- We should be evaluating AI systems, not just models, as the real-world impact of AI emerges from interfaces, data pipelines, human interaction, and policies working together.
- AI Evaluation is difficult: unlike traditional software, AI is probabilistic, data-dependent, and prone to edge-case breakdowns and unforeseen behaviors.
- Effective evaluation is deliberate and structured, and every project should iterate through six core activities:
- framing and scoping
- choosing metrics
- assembling evaluation data
- running and computing
- interpreting and communicating
- deciding and acting
- Much of AI evaluation can be automated, but the most critical decisions, such as what to measure, how to weigh trade-offs, and what actions to take, still require human judgment.
Evaluating AI Systems ebook for free