5 Counterfactual evaluations
Counterfactual evaluations provide a way to ask “what if” questions about AI decisions using historical production data rather than live experimentation. The chapter begins by connecting this idea to causal inference, which focuses on cause and effect instead of simple correlation. It explains that these methods estimate how a different model, ranking policy, recommendation strategy, or agent action might have performed, but only as an estimate built from observed logs, propensities, outcomes, and assumptions.
To make counterfactual evaluation work, the chapter emphasizes the need for rich, intentional logging and a clear evaluation structure. At minimum, systems should record the decision context, the action taken, the probability of that action, the candidate set, the model or policy version, and the resulting outcome. With that foundation, methods such as inverse propensity weighting, off-policy evaluation, and policy value estimation can compare a candidate policy against the logged baseline. The text also highlights that usefulness depends on overlap between policies, accurate propensities, enough contextual detail, and stable effective sample size.
The chapter closes by stressing both the strengths and the limits of this approach. Counterfactual evaluations are fast, low-risk, and useful when A/B tests are impractical, especially for rare or high-stakes scenarios, but they are only as trustworthy as the data and assumptions behind them. Common failures include incomplete logging, biased probabilities, poor overlap, and missing context. The discussion extends these ideas to search, recommendation, LLM, and agentic systems, where careful engineering is needed to balance logging quality, storage cost, latency, and privacy while preserving enough evidence to support reliable offline judgment.
An input-output framework for counterfactual evaluations, illustrating how logged data feeds into processes like off-policy evaluation to generate actionable metrics such as policy value, incremental action value, and uplift.
illustrates a simplified counterfactual evaluation workflow. Logged interaction data is reweighted using an off-policy correction method, such as inverse propensity weighting, to estimate the value of a candidate policy. That estimate is then compared with a baseline policy. The workflow assumes correct logging, sufficient overlap between the logged and candidate policies, and enough contextual information to support meaningful comparison.
illustrates an example of policy value evaluation in a movie recommendation system, illustrating how logged data, such as recommended movies, alternative options, and contextual details, is processed using techniques like inverse propensity weighting to produce actionable metrics such as engagement and retention.
Summary
- Counterfactual evaluations help estimate what might have happened if a different model, ranking policy, recommendation policy, or decision strategy had been used.
- These evaluations are useful, but they are estimates under assumptions, not perfect replays of an alternate reality.
- Causal inference is about understanding cause and effect. Counterfactual thinking is one way to reason about the outcomes of decisions we did not actually take.
- Counterfactual logging captures the decision as it looked at the time: the context, candidate set, selected action, action probabilities or scores, policy version, and observed outcome.
- Policy value is the expected performance of a policy. Off-policy evaluation is the family of methods used to estimate that value from data collected by a different policy.
- Inverse Propensity Weighting reweights logged outcomes based on how likely the new policy would have been to take the logged action compared to the original policy.
- IPW and related methods require accurate propensities, enough overlap between the logged and new policies, and careful checks for unstable weights.
- Effective sample size, confidence intervals, segment-level checks, and weight distributions help determine whether a counterfactual estimate is trustworthy.
- Good logging architecture balances evaluation needs with latency, storage cost, privacy, governance, and long-term maintainability.
AI Model Evaluation ebook for free
