8 Direct Alignment Algorithms (DAAs)
Direct alignment algorithms are a simpler way to optimize language models for human preferences without training an intermediate reward model or running an online reinforcement learning loop. They use the same preference data as RLHF, but turn the problem into a direct optimization objective that is easier to implement, cheaper to train, and often more practical for experimentation and deployment. The chapter emphasizes that this simplification is the main reason these methods have become so widely used.
The central example is Direct Preference Optimization, which is derived from the standard RLHF objective and a Bradley-Terry model of preferences. The key idea is that the model learns to increase the relative likelihood of chosen completions over rejected ones, compared with a reference model, with the strength of the update controlled by a KL-related parameter. From this derivation comes a closed-form optimal policy and an equivalent loss that is just a negative log-sigmoid over the difference in log-probability shifts, making the method mathematically elegant and straightforward to implement in practice.
The chapter also surveys limitations and variants. Because DAA methods usually treat preference pairs uniformly and rely on offline data, they can suffer from overfitting, preference displacement, and a lower performance ceiling than online RL methods in some settings. Variants such as cDPO, IPO, ORPO, SimPO, REBEL, and online extensions try to address these weaknesses by changing the loss, adding robustness to noisy labels, reducing memory use, or bringing in fresh on-policy data. The practical takeaway is that DAAs are an excellent starting point for preference tuning, but data quality, data provenance, and the choice between offline simplicity and online adaptability remain crucial.
Sketch of preference displacement in DPO.
Summary
- Direct Alignment Algorithms (DAAs) solve the same RLHF objective as policy gradient methods but without training a reward model or running RL – making them far simpler to implement and cheaper to run.
- DPO, the most prominent DAA, derives a closed-form loss from the Bradley-Terry preference model by showing that the optimal policy implicitly defines a reward function. The loss increases the margin between chosen and rejected completions, weighted by how wrong the implicit reward currently is.
- A known problem with DPO is preference displacement: the optimization reduces the probability of both chosen and rejected responses, with rejected decreasing more. This, combined with the use of offline data, is a posited reason why RL-based methods tend to slightly outperform DAAs in peak possible performance.
- Many variants address DPO’s limitations – IPO and cDPO handle label noise, SimPO adds length normalization, ORPO removes the reference model, and online variants like Online DPO generate fresh completions during training.
- Despite the performance gap with RL methods, DAAs remain widely used in leading models due to their simplicity, and are the recommended starting point for most practitioners doing preference fine-tuning with synthetic data.
Reinforcement Learning from Human Feedback ebook for free
