15 Regularization

The chapter explains that regularization is essential in RLHF because powerful post-training updates can push a model too far from its strong reference model and cause over-optimization. When this happens, the model may start producing degraded or nonsensical outputs such as repeated text, strange characters, or broken reasoning. Regularization is therefore used to keep optimization controlled while still allowing the model to improve from human feedback and other reward signals.

The central technique is KL-based control, where the reward objective is reduced by a penalty for drifting from a reference policy. In practice, this penalty is usually computed between the current RL policy and a fixed reference model, often using a reverse KL form that strongly discourages assigning probability to outputs the reference model considers unlikely. The chapter also shows how the computation is approximated efficiently with log probabilities, and notes that although many regularization tricks appear in specific systems, KL regularization remains the core stabilizing tool in most RLHF implementations.

The chapter also argues that RL has an implicit regularizing effect beyond any explicit penalty. Empirical results suggest that RL generalizes better than SFT and forgets less because on-policy updates stay close to the model’s own distribution, while SFT trains from a fixed external dataset that can pull the model away from prior knowledge. This difference is tied to forward versus reverse KL behavior: SFT tends to spread probability broadly and can overwrite earlier capabilities, whereas RL updates are more local and preserve existing structure. The chapter closes by surveying other stabilizers such as pretraining-gradient replay, NLL terms added to DPO, and margin-based losses for reward modeling.

Forgetting dynamics for forward KL (SFT) versus reverse KL (RL). The “old” mode represents prior knowledge, the “new” mode represents the target task. Forward KL stretches the policy to cover the target and pulls mass away from the old mode (top right), while reverse KL shifts the new mode toward the target without disturbing the old mode (bottom right). From Chen et al. 2025, with permission of the author.

Bias toward KL-minimal solutions reduces forgetting. (Left) Among policies that solve the new task, RL converges to those closest in KL to the base model. (Right) This KL bias yields higher prior-task retention at matched new-task performance compared to SFT. From Shenfeld, Pari, and Agrawal 2026. License CC-BY.

• Regularization in RLHF prevents the optimizers introduced in Chapter 6 from pushing the model too far from its starting point, avoiding degenerate outputs like repeated text, language switching, or nonsensical reasoning.
• The primary regularization tool is a KL divergence penalty between the current policy and a reference model, applied to generated tokens during training. The reverse KL direction used in most RLHF implementations strongly penalizes the new model for placing probability mass where the reference model assigns low probability.
• Other regularization approaches include mixing in pretraining gradients or a negative log-likelihood (NLL) term on winning completions to preserve general language modeling quality, and margin losses for reward model training (as in Llama 2) or reward-weighted DPO variants that integrate reward scores into the preference optimization update.
• RL-based post-training also provides implicit regularization: RL generalizes where SFT memorizes (due to reverse vs. forward KL dynamics), and on-policy sampling inherently biases RL toward solutions that stay close to the base model, reducing forgetting of prior knowledge.

FAQ