AI Model Evaluation you own this product

Leemay Nassery

MEAP began August 2025
Last updated September 2026
Publication in January 2027 (estimated)

ISBN 9781633435674
250 pages (estimated)

Included with a Manning Online subscription

printed in black & white

available in Russian

catalog / Data Science / Data Analysis / Optimization and Experimentation

resources: Source code Book forum Source code on GitHub

table of content

Part 1: AI Model Offline Evaluations

1 Setting the stage for offline evaluations

1.1 Shifting AI enthusiasts to evaluation enthusiasts

1.2 Model product development lifecycle

1.3 What are AI model offline evaluations?

1.3.1 The "offline" in offline evaluations

1.3.2 Offline evaluations for internal tools

1.3.3 Data in its many forms

1.3.4 Offline metric categories

1.3.5 Many metrics to choose from

1.4 The two layers of offline evaluations

1.5 AI v.s heuristics

1.6 Influencing online controlled experiments

1.7 Practical applications of offline evaluations

1.7.1 Offline evaluations for production observability

1.7.2 Online-offline correlation

1.8 When not to use offline evaluations

1.8.1 Feedback loop dynamics

1.8.2 Balancing offline and online approaches

1.8.3 UX considerations

1.8.4 Cross domain collaboration is key

1.9 Summary

2 Anatomy of an offline evaluation

2.1 The many faces of offline evaluations

2.1.1 Performance evaluations

2.1.2 Diagnostic evaluations

2.1.3 Simulation-based evaluations

2.1.4 Combining evaluations to bridge the gap

2.1.5 How evaluation feeds development cycles

2.2 Anatomy of an offline evaluation

2.2.1 Data as input

2.2.2 Designing the offline evaluation

2.2.3 Movie recommendation example

2.3 Common pitfalls of offline evaluations in real world settings

2.3.1 Lack of trust

2.3.2 Limited guidance on acceptable outcomes

2.3.3 Friction with production systems

2.3.4 Evaluations that don’t generalize

2.3.5 Overfitting historical data

2.3.6 Difficulty in interpreting metrics

2.3.7 Difficulty in interpreting metrics

2.4 Engineering considerations

2.4.1 Scaling offline evaluations across teams

2.5 Summary

3 Using offline evaluations as diagnostics

3.1 Diagnosing model behavior

3.1.1 Evaluating bias in movie recommendations

3.1.2 How diagnostics complement performance offline evaluations

3.1.3 Bridging the gap between model and product level understanding

3.2 Show me the metrics

3.2.1 Error analysis metrics

3.2.2 Fairness and bias evaluation metrics

3.2.3 Robustness testing metrics

3.3 Connecting diagnostics to product impact

3.3.1 How product leads can use diagnostic metrics for decision making

3.3.2 Balancing the product experience

3.4 Practical applications of diagnostic evaluations

3.4.1 Testing model robustness against noise

3.4.2 Diagnosing data distribution shifts

3.4.3 Improving cold-start cases

3.4.4 Detecting user segment disparities

3.4.5 Diagnosing LLM and agent failures

3.4.6 Making you think about the product experience

3.5 Common challenges

3.6 What makes a good diagnostic offline evaluation?

3.6.1 Have a goal in mind

3.6.2 Have the right data

3.6.3 Choose the right comparison point

3.6.4 Make sure you can take action

3.6.5 Repeat, Repeat, Repeat

3.7 Engineering considerations

3.7.1 Fine-grained, detailed, logging

3.7.2 Tooling to improve interpretability

3.7.3 Sensitive data

3.8 Summary

4 Engineering system performance evaluations

4.1 Why latency and load time really matter

4.1.1 Illustrating the impact on user experience and business metrics

4.1.2 Technical constraints and scaling challenges

4.2 Engineering system performance metrics

4.2.1 Key load time metrics

4.2.2 Key latency metrics

4.2.3 LLM and agent-specific performance metrics

4.2.4 Combining load time and latency metrics for performance evaluations

4.3 Shadow traffic

4.3.1 How shadow traffic works

4.3.2 Shadow traffic versus A/B testing

4.3.3 Benefits and challenges of shadow traffic

4.3.4 Mimicking shadow traffic

4.4 Latency degradation experiments

4.4.1 4.4.1 Designing latency degradation tests

4.4.2 Key online evaluation system performance metrics

4.4.3 Movie recommendations example

4.5 Engineering considerations

4.5.1 4.5.1 Make sure to capacity plan your infrastructure resources

4.5.2 4.5.2 You’ll never regret time spent testing under realistic product conditions

4.5.3 4.5.3 Keep in mind these performance optimization tactics

4.6 Summary

5 Counterfactual evaluations

5.1 What is causal inference

5.2 Counterfactual evaluations

5.2.1 Anatomy of a counterfactual evaluation

5.2.2 The assumptions hiding underneath counterfactual evaluations

5.2.3 Why counterfactual evaluations matter

5.3 What is counterfactual logging?

5.3.1 How counterfactual logging relates to causal inference

5.3.2 How much data is "enough"?

5.3.3 Data alone is not enough

5.4 Estimating outcomes

5.4.1 Policy value

5.4.2 Off-policy evaluations

5.4.3 Designing incremental action value for your domain

5.5 Logging best practices

5.5.1 Log all possible actions

5.5.2 Record action probabilities

5.5.3 Include contextual metadata

5.5.4 Consistent and unbiased logs are the best types logs

5.5.5 Practical example: search engine

5.6 Strengths and realities of counterfactual evaluations

5.7 Common pitfalls in counterfactual logging

5.7.1 Incomplete data logging

5.7.2 Biased propensity scores

5.7.3 Poor overlap between the logged policy and new policy

5.7.4 Ignoring contextual variables

5.8 Policy value evaluation for movie recommendations

5.8.1 The setup: data, design, and metrics

5.8.2 Challenges along the way

5.8.3 Outcomes and insights

5.8.4 Making it actionable

5.9 LLM and agentic systems are tricker

5.9.1 Why the action space gets messy fast

5.9.2 Start with discrete, loggable decision points

5.9.3 A practical example: an agent with tool routing

5.10 Engineering considerations

5.10.1 Balancing system performance and logging

5.10.2 Massive amounts of data

5.11 Summary

Part 2: AI Model Online Evaluations

6 Evaluating models in an A/B test

6.1 A/B testing

6.1.1 Why A/B test

6.2 The 'right time' to A/B test a model

6.2.1 Model maturity

6.2.2 Timelines

6.2.3 How offline evaluations influence A/B test readiness

6.3 Designing an A/B test

6.3.1 Operational setup

6.3.2 Experimental design

6.3.3 Hypothesis and purpose

6.3.4 Model purpose & intent

6.3.5 Experiment validity checks before interpreting results

6.3.6 Movie recommendation A/B test example

6.4 Prerequisites for running A/B tests

6.4.1 Latency monitoring

6.4.2 Feature drift monitoring

6.4.3 Logging still matters

6.5 Quirks of A/B testing models

6.5.1 Multiple model variants increase complexity

6.5.2 Novelty effect

6.5.3 Model warm-up and cold start effects

6.5.4 Variance and sensitivity to noise

6.5.5 Offline and online mismatch

6.6 Interpreting ambiguous or low-signal results

6.7 Engineering considerations

6.7.1 Infrastructure-induced latency or errors

6.7.2 If you just don’t have an A/B testing platform

6.8 Summary

7 From offline evaluation to live experiment

7.1 Model selection playbook

7.2 Internal beta testing before exposing to users

7.3 Model selection scoring rubric

7.4 Model maturity nuances in selection decisions

7.5 Stakeholder alignment and communication

7.6 Mapping offline insights into targeted online monitoring

7.6.1 Translating offline evaluations into online watch points

7.6.2 Building a metric mirror table

7.6.3 Tracking correlation between offline and online metrics over time

7.7 Interim checks

7.8 Engineering considerations

7.8.1 Realtime monitoring tied to product and business metrics

7.8.2 Side-by-side model comparison tools

7.8.3 Reliable infrastructure for iterative improvement

7.9 Summary

8 Pitfalls of online metrics

8.1 Choosing metrics that actually matter

8.1.1 Weak metrics waste the offline investment

8.1.2 Contract between product strategy and AI model evaluation

8.2 Feedback loops unique to AI models

8.2.1 How to combat feedback loops in evaluations

8.3 Metric gaming and model exploitation

8.3.1 Reward hacking in AI models

8.3.2 Reward hacking isn’t limited to models

8.3.3 Why this risk is sharper in AI A/B tests

8.3.4 Combating reward hacking and exploitation

8.4 The blind spots of online metrics

8.4.1 Fairness across segments

8.4.2 Robustness in edge cases

8.4.3 Qualities like trust, coherence, or novelty that resist simple measurement

8.5 The importance of the right metrics framework for AI

8.5.1 Building metric hierarchies

8.5.2 Revisiting metrics as models and products evolve

8.5.3 Aligning evaluation targets with both product and model goals

8.6 Statistical refinements for AI online metrics

8.6.1 Top-line averages may misrepresent specific user groups

8.6.2 Variance reduction techniques (CUPED, regression adjustment)

8.6.3 Pitfalls of covariate choices

8.7 Engineering considerations

8.7.1 Logging and attribution in ensemble or blended models

8.7.2 Monitoring for metric drift in retraining pipelines

8.7.3 Alert fatigue

8.7.4 Guarding against variance inflation

8.8 Summary

Part 3: LLM-as-a-judge evaluations

9 LLM-as-a-judge fundamentals

9.1 The case for LLM-as-a-judge

9.2 When deterministic metrics aren’t enough

9.3 What LLM judges are good at and what they are not

9.4 Getting the goal and data right

9.4.1 Defining the goal

9.4.2 Designing the data

9.5 How to design a good LLM-as-a-judge evaluation

9.5.1 Defining the judging task

9.5.2 Designing the prompt

9.5.3 Choose the right judge format: scoring, ranking, or pairwise comparison

9.5.4 Selecting the right evaluator model

9.6 Prompt evaluation feedback loop

9.6.1 LLM-as-a-judge for the movie recommender model

9.7 Common LLM judge failure modes

9.7.1 Position bias

9.7.2 Verbosity bias

9.7.3 Style and self-preference bias

9.7.4 Context insufficiency

9.7.5 Prompt injection against the judge

9.7.6 Overconfidence and fluent explanations

9.8 Validating an LLM-as-a-judge evaluation

9.9 When not to use LLM-as-a-judge

9.10 Engineering Considerations

9.10.1 Prompt debugging 101

9.10.2 Document everything, just as you would any other code base

9.10.3 Calibrate before increasing scale

9.10.4 Observability isn’t optional, it’s a must.

9.10.5 Engineering factors influencing LLM model selection

9.11 Summary

10 LLM-as-a-judge in production

10.1 Panels and ensembles of judges

10.1.1 It’s OK to disagree

10.1.2 When is a panel of judges worth it?

10.2 Decomposed and checklist-based judging

10.2.1 Why checklists could be better than 1–5 scoring on certain tasks

10.2.2 Writing checklist items

10.2.3 Checklists really make debugging easier

10.3 Reasoning patterns: chain-of-thought, debate, and critique

10.3.1 Explicit reasoning traces before verdict

10.3.2 Self-critique loops

10.3.3 Debate-style judging (argue-then-judge prompting)

10.4 Self-consistency and stability testing

10.4.1 Consistent doesn’t mean correct and correct doesn’t mean consistent

10.4.2 Repeated Sampling

10.4.3 Swapped-order reruns

10.4.4 Variance as a primary metric

10.5 Escalation, abstention, and tiered judging

10.5.1 Tiered Judging

10.5.2 Escalation

10.5.3 Abstain or defer scoring

10.6 When evaluation fails: postmortems and recovery

10.6.1 Quiet failures

10.6.2 Triage: Figure out what actually broke

10.6.3 Run a post mortem / incident review

10.6.4 Rebuilding trust in the evaluation system

10.7 Summary

Appendix

Appendix A: Appendix

A.1 Github code repository

Overview

7 From offline evaluation to live experiment

The chapter explains that A/B testing should be treated as one part of a larger evaluation system rather than a standalone launch tactic. To move models from offline work into live experiments responsibly, teams need a model selection playbook that clarifies the user problem, summarizes offline wins and weaknesses, records risks from sandbox or beta testing, defines online watchpoints, and states the decision rule for shipping, iterating, rolling back, or retesting. This keeps online testing focused on strong, well-motivated variants instead of wasting time and resources on models that are not ready or not meaningfully different.

A major theme is the value of internal beta testing before exposing users to a model. Employee testing in a controlled environment can uncover obvious defects, skewed rankings, safety issues, and operational problems that might not appear in offline metrics alone. The chapter uses a recommendation example to show how internal reviewers can catch a hidden popularity bias, where one title repeatedly dominates top results, allowing the team to adjust model weights, add diversity constraints, and re-run offline checks before any live rollout. The chapter also stresses that beta feedback is only a readiness signal and must be structured carefully so it is useful rather than noisy.

Finally, the chapter argues that offline findings should directly shape online monitoring and future decisions. Teams should translate known offline failure modes into specific live watchpoints, build metric mirror tables that connect offline metrics to their online proxies, and maintain a rolling dataset that tracks how offline scores relate to online outcomes over time. This lets teams identify which offline metrics are actually predictive, refine the scoring rubric, and set better thresholds for future tests. The overall message is that the transition from offline evaluation to live experimentation is a continuous, well-instrumented loop involving engineering, communication, and learning, not a one-time handoff.

illustrates an example of a systemic ranking bias caught during internal beta testing: The same popular title (represented as a star) repeatedly appears in the top position of the 'Movies To Watch' row across different users and segments. This pattern, observed before launch, indicates an overweighted popularity feature in the re-ranking stage which is a problem that could influence engagement metrics if not caught before.

illustrates the offline to online correlation feedback loop.

Summary

Before exposing a model to real users, enable a sandbox environment for employees to compare model variants to identify issues that may have gone unnoticed during the offline evaluations stage.
Create a model selection playbook for deciding which models to include in an A/B test. Use a weighted scoring rubric: offline performance, product heuristics, risk, strategic alignment.
Model maturity influences both selection playbook thresholds and in-test monitoring intensity.
Stakeholder alignment matters because building AI models is costly, the details matter, and you don’t want to A/B test something that’s out of sync with product requirements or stakeholder expectations.
Identify offline weaknesses and turn them into specific live monitoring points during the A/B test (i.e. bias gaps, recall drops).
Use concrete examples that correlate to product requirements to guide online experimentation monitoring, like tracking diversity per segment online if it dropped offline, with clear alert thresholds.
Maintain a metric mirror table that maps each offline metric to its online equivalent, including formulas and alert triggers.
Continuously track how offline metrics correlate with online results and adjust your model selection rubric to prioritize metrics with stronger predictive power.
Invest in building side-by-side model comparison tools to use during beta testing.
Offline evaluation weaknesses (like bias, diversity drops, or latency risks) should translate into live A/B test watchpoints with alerting thresholds. Build metric mirrors that pair each offline metric with an online counterpart, then track correlations over time to refine which offline metrics are the best predictors of real-world outcomes

FAQ

What is a model selection playbook, and why do teams need one before running an A/B test?

A model selection playbook is a lightweight rubric for deciding which model variants are ready to enter online evaluation. It helps teams avoid wasting test capacity, misaligning with stakeholder expectations, and testing variants without a clear hypothesis, known risks, and monitoring plan.

What core components should a model selection playbook include?

A solid playbook should include an offline evaluation summary, sandbox or beta testing feedback, and a model selection rubric that brings together metrics, product heuristics, operational readiness, and other decision factors.

What questions should a model selection playbook answer before a model goes into an A/B test?

It should answer: what problem the model is solving, which offline metrics improved or degraded, what risks were found, what online watchpoints will monitor those risks, and what the decision rule is for shipping, iterating, rolling back, or retesting.

Why is internal beta testing useful before exposing a model to real users?

Internal beta testing lets employees stress-test the model, uncover obvious failure modes, and catch edge cases in a low-risk environment. It is best for finding defects and qualitative issues, not for estimating user impact.

How should feedback from internal beta testers be interpreted?

It should be interpreted carefully because employees are often not representative of real users. They may be more technical, more familiar with the product, and more likely to test unusual scenarios, so their feedback is useful for readiness and bug detection, not for declaring success.

What was the “Batman problem” example meant to show?

It showed how a recommendation model can over-weight popularity and repeatedly surface the same title in top positions, even when it is not relevant to the user. Beta testing caught the issue early, allowing the team to adjust feature weights and add diversity constraints before launch.

What does a model selection scoring rubric do?

The scoring rubric consolidates offline performance, product heuristics, model maturity, strategic alignment, operational readiness, and sandbox feedback into one transparent decision framework so that model advancement is repeatable and defensible.

How does model maturity affect selection decisions?

Model maturity changes how strict the selection thresholds should be. Early-stage models may get more flexibility on trade-offs like latency or cost, while mature models usually need stronger evidence and more stringent readiness checks before being fast-tracked to production.

How do offline evaluation results become online watchpoints?

Teams identify offline failure modes, map them to equivalent live metrics, and set alert thresholds tied to known risks. This creates a metric mirror table that links offline metrics to online monitoring, helping catch problems early in the experiment.

Why track correlation between offline and online metrics over time?

Tracking correlation helps teams learn which offline metrics are actually predictive of online success. By maintaining a rolling dataset and updating correlations after each experiment, teams can weight their selection criteria based on real historical evidence instead of assumptions.

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

pdf, ePub, online

$47.99 $23.99

you save $24.00 (50%)

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

$47.99 $23.99

you save $24.00 (50%)

eBook

pdf, ePub, online

$47.99 $23.99

you save $24.00 (50%)

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more