Evaluation in RLHF is the set of methods used to judge how training choices affect a model’s quality, and the chapter treats benchmarks as the main public expression of that process. It emphasizes that evaluation is not fixed but evolves alongside modeling goals: what gets measured tends to mirror the current priorities of training and post-training. As language models have advanced, evaluation has shifted from narrow chat-oriented tests to broader suites covering knowledge, reasoning, math, coding, safety, and increasingly difficult tool-use and reasoning tasks.
A central theme is that prompting format can change scores dramatically, sometimes turning strong performance into apparent failure if the format is mismatched. The chapter traces this progression from few-shot prompting, to zero-shot instruction following, to chain-of-thought prompting and generation-based evaluation, showing how models became stronger and more open-ended in their responses. It also explains the tradeoff between sampling-based exact match and log-likelihood scoring, and why modern reasoning models often require carefully designed prompts, answer formats, and decoding settings to reveal their true capability.
The chapter argues that many external comparisons are unreliable because labs use different internal prompts, sampling setups, inference-time scaling, hidden test selections, and other undocumented choices. Benchmark results are further confounded by saturation, where popular tests quickly become too easy to distinguish models, and by contamination, where training data overlaps with evaluation data and inflates apparent performance. To make evaluation more useful, labs increasingly rely on internal, task-specific suites and open tooling, but the overall message is that robust evaluation remains difficult, noisy, and highly sensitive to setup details.
Report from Epoch AI showing how major AI evaluations are rapidly saturated over time (saturation is when a given benchmark reaches full performance and models no longer have meaningful signal). License CC-BY.
Summary
Evaluation for post-training has evolved through distinct phases: early chat-focused benchmarks scored by LLM-as-a-judge (MT-Bench, AlpacaEval), multi-skill suites covering knowledge, math, code, and safety, and the current era of challenging reasoning and tool-use tasks (GPQA, SWE-Bench, AIME).
Prompting format dramatically affects benchmark scores – the field has progressed from few-shot prompting through zero-shot to chain-of-thought generation, and mismatches between evaluation format and how the model was trained can cause performance to drop to near zero.
Evaluation comparisons across labs are unreliable because each uses different prompts, sampling parameters, and internal evaluation sets. Scores should be treated as rough indicators with meaningful error bars rather than precise rankings.
Dataset contamination (where evaluation data leaks into training through web scraping or user queries) is a persistent confounding factor, and decontamination via n-gram overlap matching is now standard practice in responsible post-training pipelines.
FAQ
What is the main purpose of evaluation in RLHF and post-training?Evaluation is used to measure the quality and influence of training, usually through benchmarks. It helps researchers understand whether training improved the model on the desired skills and where it still fails.How did RLHF evaluation evolve over time?It moved through several phases: early chat-focused benchmarks, then multi-skill suites covering knowledge, reasoning, math, coding, instruction following, and safety, and most recently reasoning- and tool-focused evaluations for harder problems.Why did early chat benchmarks like MT-Bench and AlpacaEval rely on LLM-as-a-judge?They used LLMs such as GPT-4 to replace human evaluators because it was a cheaper and more scalable way to approximate human preference judgments.What is length bias in evaluation?Length bias is when longer responses get higher scores even if they are not better. This became a known issue in some LLM-as-a-judge benchmarks and led to length-controlled variants.Why does prompting format affect benchmark performance so much?Small changes in prompt wording, answer format, or output constraints can dramatically change scores. A model may appear much weaker or stronger depending on whether the evaluation format matches how it generates answers.What is the difference between exact match and log-likelihood scoring?Exact match checks whether the model’s generated answer is correct, often by sampling or majority vote. Log-likelihood scoring compares which answer option the model assigns the highest probability to, which is more common in pretraining-style evaluation.Why are many external model comparisons unreliable?Different labs use different hidden prompts, sampling settings, system messages, and internal test sets. Because of that, published scores often have large error bars and cannot be compared as if they were perfectly controlled experiments.What is benchmark saturation?Benchmark saturation happens when a benchmark becomes too easy and scores approach 100%. At that point, the benchmark stops providing meaningful signal for measuring model progress.What is dataset contamination, and why does it matter?Dataset contamination happens when evaluation data leaks into training data, intentionally or unintentionally. It can inflate benchmark scores and make it hard to know whether a model truly generalized or merely memorized the test set.What evaluation tools are commonly used in the open-source ecosystem?Examples include Inspect AI, LightEval, Eleuther AI’s evaluation harness, AI2’s OLMES-based library, HELM, and Mosaic/Databricks Eval Gauntlet. These tools help standardize and automate model evaluation.
pro $24.99 per month
access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
Reinforcement Learning from Human Feedback ebook for free
