3 Picking evaluation metrics

The chapter explains that evaluation metrics are the tools that turn abstract goals into measurable evidence, but no single metric can capture everything that matters. Because metrics are always tied to assumptions, inputs, computation methods, outputs, and interpretation, choosing them deliberately is crucial for making sound decisions. The core message is to avoid picking metrics simply because they are popular, easy to compute, or familiar, and instead assemble a set of measures that actually fits the task, the context, and the stakeholders who will use the results.

It distinguishes several major metric types and when each is useful. Automatic metrics are fast and scalable but may capture only proxies of quality; judgment-based metrics can better reflect contextual or normative concerns but are slower, costlier, and less scalable; reference-based metrics are easy to compare and benchmark but depend on high-quality ground truth; reference-free metrics work when no single correct answer exists; and execution-based metrics assess end-to-end task completion in realistic workflows. The chapter also stresses that metric names alone are not enough, since different implementations of the same metric can encode very different assumptions.

To judge whether a metric is worth trusting, the chapter lays out key quality criteria: construct validity, causal proximity, movability, reliability, robustness, correlation with human judgments, interpretability, and measurability. A good metric should measure the right thing, change for the right reasons, remain sensitive to improvement, produce consistent results, withstand small irrelevant variations, align with expert judgment when appropriate, be understandable to decision-makers, and be feasible to compute. When off-the-shelf metrics do not fit, the chapter recommends designing custom metrics, and it closes by emphasizing that metrics must be versioned, revisited, and maintained over time as systems, users, and goals evolve.

Core elements of an evaluation metric. Every metric transforms inputs into an output through a computation method, but its value depends equally on how the resulting score is interpreted and used in decision-making

Desirable qualities of evaluation metrics. While no metric is perfect, the more of these qualities a metric exhibits, the more confidence you can place in the conclusions it supports.

Designing custom metrics when existing metrics are insufficient. Starting from an evaluation goal and construct of interest, the process progressively translates abstract evaluation requirements into a measurable and actionable metric.

• Evaluation metrics are measurements that act as operational proxies for decisions, and every metric encodes assumptions about what matters, how it should be measured, and how results should be interpreted.
• No single metric is sufficient. Effective evaluation relies on selecting multiple complementary metrics that collectively capture different aspects of system behavior and failure modes.
• You must assess metric quality explicitly. Criteria such as construct validity, causal proximity, movability, reliability, robustness, correlation with human judgment, interpretability, and measurability help you distinguish metrics that will genuinely help you from those that might distract or mislead you.
• Don’t hesitate to create custom metrics when off-the-shelf metrics are unable to capture your evaluation requirements. Designing a custom metric requires clearly defining the target construct, choosing appropriate output formats, specifying computation procedures, and piloting behavior, and thoroughly documenting assumptions and intended use.

FAQ