Overview

1 The importance of AI evaluation

AI evaluation is presented as the disciplined process of measuring how an AI system behaves in the real context where it will be used, not just how well a model scores on a benchmark. The chapter opens with examples of costly failures in pricing, hiring, and customer service to show that AI systems can create serious financial, legal, and social harm after deployment. These incidents illustrate why evaluation must look beyond simple accuracy and ask whether outputs are trustworthy, useful, safe, and appropriate for the people and institutions affected.

The chapter also clarifies what counts as an AI system and why that matters for assessment. An AI system is defined by what it is intended to do, whether it predicts, generates, or takes actions, and the text emphasizes that a model is only one component of a larger product that also includes interfaces, workflows, data pipelines, and human oversight. Because users interact with the full system rather than the model alone, evaluation has to account for the whole experience and for the fact that many real-world systems are hybrid, combining predictive, generative, and agentic elements.

Finally, the chapter explains why AI evaluation has become both essential and difficult. Modern AI is more capable, more widely deployed, and more opaque than traditional software, yet current evaluation practices often remain shallow, late-stage, and overly dependent on limited metrics or unrepresentative data. To address this gap, the chapter recommends a systematic, iterative approach: define the goal and scope, choose the right metrics, gather suitable data, compute results carefully, interpret them in context, and turn findings into decisions. The overarching message is that reliable AI requires continuous evaluation supported by human judgment, not one-time testing before launch.

The difference between an AI model and an AI system. A model is a self-contained computational component that transforms inputs into outputs, while an AI system combines the model with s user interfaces, data pipelines, integration logic, human-in-the-loop mechanisms, and other functionalities and features to perform real-world tasks.
The difference between an AI model and an AI system. A model is a self-contained computational component that transforms inputs into outputs, while an AI system combines the model with s user interfaces, data pipelines, integration logic, human-in-the-loop mechanisms, and other functionalities and features to perform real-world tasks.
Three broad categories of AI systems: predictive systems that make forecasts or classifications, generative systems that create content, and agentic systems that take actions to achieve goals in an environment.
Three broad categories of AI systems: predictive systems that make forecasts or classifications, generative systems that create content, and agentic systems that take actions to achieve goals in an environment.
The six main steps of an AI evaluation project, typically executed in an iterative fashion. All six are essential, and their order matters.
The six main steps of an AI evaluation project, typically executed in an iterative fashion. All six are essential, and their order matters.

Summary

  • AI capabilities have surged while evaluation practices lag, and high-profile failures show the cost of that gap.
  • A widening capability–comprehension gap creates two opposing risks: over-reliance and under-reliance.
  • AI evaluation means systematically assessing how systems perform under real-world complexity and unpredictability.
  • We should be evaluating AI systems, not just models, as the real-world impact of AI emerges from interfaces, data pipelines, human interaction, and policies working together.
  • AI Evaluation is difficult: unlike traditional software, AI is probabilistic, data-dependent, and prone to edge-case breakdowns and unforeseen behaviors.
  • Effective evaluation is deliberate and structured, and every project should iterate through six core activities:
    • framing and scoping
    • choosing metrics
    • assembling evaluation data
    • running and computing
    • interpreting and communicating
    • deciding and acting
  • Much of AI evaluation can be automated, but the most critical decisions, such as what to measure, how to weigh trade-offs, and what actions to take, still require human judgment.

FAQ

What does AI evaluation mean in this chapter?AI evaluation is the systematic practice of understanding and measuring how an AI system behaves in its intended context of use. It goes beyond benchmark scores or accuracy alone and asks whether the system reliably achieves its purpose in real-world conditions.
Why is it important to evaluate the full AI system instead of only the model?The model is only one part of an AI product. The full system also includes interfaces, workflows, data pipelines, integration logic, and human oversight. A strong model can still perform poorly in a badly designed system, so evaluation should focus on the complete user-facing system.
What are the three broad categories of AI systems mentioned in the chapter?The chapter groups AI systems into three categories: predictive systems, which forecast or classify; generative systems, which create content; and agentic systems, which take actions to achieve goals in an environment.
Why are traditional software testing methods not enough for AI systems?Unlike deterministic software, AI systems are probabilistic, data-driven, and sensitive to context. They can behave differently across users, inputs, and deployment settings, so they require broader evaluation than fixed-output testing.
What makes AI evaluation especially challenging?AI evaluation is challenging because performance is context-dependent, ground truth may be missing, outputs can be open-ended, human-AI interaction affects outcomes, and system behavior can be opaque or even unpredictable. It also requires trade-offs among multiple quality dimensions such as fairness, robustness, and explainability.
Why can benchmark scores give a false sense of confidence?Benchmarks often cover only a narrow slice of real-world use. They may be outdated, biased, or unrepresentative of production conditions, so a good score on a benchmark does not guarantee good performance in practice.
What are the main risks of under-evaluating or over-trusting AI systems?Over-reliance can lead to performance failures, ethical problems, unfair outcomes, and real-world harm when systems are deployed without enough oversight. Under-reliance can cause teams to ignore useful systems because uncertainty makes them seem less valuable than they are.
What are the six main steps in a systematic AI evaluation project?The six steps are: define the evaluation frame and scope, select metrics, acquire evaluation data, compute the metrics, interpret and communicate the results, and decide on next steps.
Why must AI evaluation be iterative and continuous?AI systems, data, and user needs change over time, so evaluation should not be a one-time activity before deployment. It should happen throughout the AI lifecycle, both offline and online, to catch issues as the system evolves.
What should an evaluation project focus on when deciding whether an AI system is fit for purpose?An evaluation should assess whether the system performs well for its intended task, in its real context of use, and across the quality dimensions that matter most. That may include correctness, robustness, fairness, privacy, security, and explainability, depending on the application and risk level.

pro $24.99 per month

  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose one free eBook per month to keep
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime

lite $19.99 per month

  • access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more


Close
choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Evaluating AI Systems Evaluating AI Systems ebook for free
Close
choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Evaluating AI Systems Evaluating AI Systems ebook for free