6 Evaluating models in an A/B test

Offline evaluation can show whether a model looks promising, but it cannot reveal its real effect on users once it is exposed in a live product. The chapter argues that AI model evaluation must include online controlled experiments because real-world behavior, latency, feedback loops, and product context can all change outcomes in ways offline metrics miss. A/B testing is framed not as a perfunctory launch check, but as a way to learn whether a model truly improves the user experience and product results.

The chapter explains that the timing and style of an A/B test should depend on model maturity, confidence in offline results, and the risks involved. Early-stage models are best tested with small, short experiments and sensitive feature-level metrics to learn whether the direction is viable, while mature models justify more rigorous tests with top-line and guardrail metrics. Strong offline evaluation lowers risk and supports earlier online testing, whereas weak offline evaluation requires tighter observability, fallback plans, and caution when interpreting results.

It also lays out the structure of a good model experiment: define the model’s purpose, form a clear hypothesis, choose appropriate variants and randomization units, set success and guardrail metrics, and verify that the experiment is valid before trusting results. The chapter emphasizes practical pitfalls such as novelty effects, cold starts, high variance, model drift, multiple comparisons, and mismatches between offline and online performance. When results are ambiguous or low-signal, the right response is to return to the original hypothesis, inspect segments and logs, and decide whether to launch, iterate, or stop based on product value rather than metric movement alone.

model ai maturity model illustrating the questions to answer for an early stage model in comparison to a model that is more mature, already in production. For an early stage model the A/B testing approach should include smaller, shorter tests measuring effect with feature-level metrics. For more mature models, the approach should include A/B tests with both top-line and feature-level metrics.

A/B testing design layers, organized from outer to inner circles. The closer to the core, the more foundational and irreversible the decisions tend to be: the outer execution layer covers operational factors like test duration, rollout timing, and sample size; the middle experimental design layer defines how the test is structured, including variants, metrics, and assignment logic; and the inner core layer captures the underlying purpose of the model and the user or product behavior it’s intended to influence.

• A/B testing is one of the most important ways product teams learn whether a change actually improves the user experience.
• When you run an A/B test depends heavily on model maturity. Early-stage models benefit from lightweight, iterative tests focused on directional insights, while mature models demand more rigorous evaluations using top-line and feature-level metrics.
• A/B test design can be thought of in layers; from flexible execution details, to structured experimental design, down to the core intent behind the model and what behavior it's meant to influence.
• The right time to run an online evaluation of a model is nuanced because it depends on the maturity of the model, timeline and offline evaluation suite.
• Ambiguous or low-signal results require careful interpretation, not just p-value hunting.
• A/B tests evaluating AI models typically have more test variants due to model tuning such as varying feature weights but this increases testing complexity (configuration wise) and demands larger sample sizes or longer test durations.
• Discrepancy between offline and online metrics can be caused by misaligned evaluation goals, stale feature usage, or inference behavior differing from training setup.
• Feature drift can cause models to perform worse over time and is one of the key reasons to monitor models once the A/B test has started.
• Model cold-start effects can skew early results; giving models time to “warm up” before interpreting metrics is often necessary. Similarly, high variance and subtle user behavior shifts mean metric selection and experiment duration must be carefully considered in AI model tests.
• Bias from training data can cause offline/online mismatch.
• The novelty effect can inflate early treatment metrics; letting an experiment run until behavior stabilizes is essential before making a decision on the effectiveness of the model being tested.
• Even without an A/B testing platform, it's still possible to evaluate online impact; it's just a lot harder and not ideal. If you don't have the capability today, then try to build a very simplistic minimal solution that resembles the beginnings of an A/B testing platform.
• Engineering considerations such as model latency and infra reliability can significantly affect user experience and, thus, test outcomes.

FAQ