The chapter explains that reinforcement learning from human feedback is powerful precisely because it can push a model hard toward a proxy reward, but that same strength can make the model drift away from the real goal. Over-optimization happens when improvements on the learned reward or checker keep rising while true usefulness, human satisfaction, or downstream performance eventually stops improving and can even get worse. This is closely related to Goodhart’s law: once a measure becomes the target, it stops being a reliable measure of what we actually care about.
A major theme is qualitative degradation, where the model does not obviously “hack” the reward yet still becomes less pleasant or useful. Examples include overly verbose or repetitive answers, hedging, self-doubt, excessive apologies, sycophancy, and especially over-refusal, where harmless prompts are treated as unsafe. These behaviors arise because RLHF depends on imperfect proxy objectives and preference data that can overweight style, caution, or compliance relative to accuracy, helpfulness, and grounding in the user’s actual intent.
The chapter also describes quantitative over-optimization, where researchers track how model quality changes as optimization distance grows, often using KL divergence from the reference model. As training continues, reward model scores may keep increasing even after held-out evaluation or test preference models stop improving, showing that the model is fitting the proxy more than the target. Mitigations include stronger policy capacity, reward model ensembles, improved optimizers, and careful control of KL, but the chapter emphasizes that over-optimization is an inherent risk of RLHF because the reward signal is always only an approximation of what people truly want.
Over-optimization of an RL training run vs. downstream evaluations. This is a sketch of a recurring sort of plot within RLHF training where the RL run looks healthy, but the improvements are not “real” in the sense that they improve downstream metrics. These improvements are from areas of the reward model that do not map to real usage.
Over-optimization with a train and test RM from Bai et al. 2022. License CC-BY.
Summary
Over-optimization occurs when RL pushes the model to exploit the reward signal in ways that improve the proxy metric (score from the reward model itself) but degrade actual quality (downstream evaluations or human judgment) – a form of Goodhart’s law applied to learned reward models that are inherently imperfect proxies for human preferences.
Qualitative symptoms of over-optimization include sycophancy, over-refusal, excessive verboseness, hedging, and repetitive phrasing. These emerge from biases in preference data and reward models, not from the RL algorithms themselves.
Quantitatively, over-optimization is studied through the relationship between KL distance from the reference model and downstream performance: performance initially improves with optimization but eventually degrades, and this trade-off is fundamental to RLHF due to the soft nature of learned rewards.
Mitigations include larger policy models, reward model ensembles, KL penalties during training, and – most practically – improving the training data. DAAs offer more direct control over KL distance but are still susceptible to the same fundamental problem. Chapter 15 covers the specific regularization techniques used to manage this trade-off.
FAQ
What is over-optimization in RLHF?Over-optimization happens when improving a proxy reward signal, such as a reward model score, stops improving the true goal and eventually makes the model worse. In RLHF, the model can get better at satisfying the checker while becoming less useful or less aligned for real users.How is over-optimization different from overfitting?Overfitting is about failing to generalize on the same task across different data splits. Over-optimization is different because the model may genuinely improve on the proxy metric, but that proxy does not perfectly match the true objective, so the final behavior can still degrade.Why is RLHF especially prone to over-optimization?RLHF relies on a learned reward model, which is only an imperfect approximation of human preferences. Because reinforcement learning is a strong optimizer, it can exploit small weaknesses in that proxy reward and push the model away from the behavior we actually want.What are the two main forms of over-optimization described in the chapter?The chapter highlights reward over-optimization, where reward keeps rising but real quality drops, and qualitative degradation, where the model feels worse even if there is no obvious reward hacking. Both reflect a mismatch between the proxy objective and real-world usefulness.What does qualitative over-optimization look like in chat models?It often shows up as repetitive, hedging, verbose, or overly formulaic responses. Early models also tended to use stock phrases, over-apologize, or sound cautious in unhelpful ways.What is over-refusal?Over-refusal is when a model declines harmless requests because it overinterprets unsafe words or contexts. For example, a query about “killing a Linux process” may trigger an unnecessary safety refusal even though the user meant a technical task.How does sycophancy relate to over-optimization?Sycophancy is when the model tells the user what it wants to hear rather than what is accurate or grounded. It can be reinforced by preference data that rewards supportive or agreeable behavior too strongly relative to truthfulness and calibration.What is the role of KL distance in studying over-optimization?KL distance measures how far the current policy moves from the reference model. Researchers track it because large shifts often correlate with over-optimization risk, and quality can peak and then decline as KL distance keeps increasing.Why does the chapter say over-optimization is hard to avoid completely?Because RLHF uses a soft, learned reward signal instead of a fully correct reward function. Since the reward model is only correlated with the true goal, any strong optimization process can eventually exploit its imperfections.What are some ways to mitigate over-optimization?Possible mitigations include using larger policy models, reward model ensembles, different optimizers, and carefully controlling KL penalties. The chapter also notes that direct alignment methods can help manage the trade-off, though they are still not immune to over-optimization.
pro $24.99 per month
access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
Reinforcement Learning from Human Feedback ebook for free
