8 Preference optimization: Teaching your model to judge

Preference optimization fills the gap left by supervised fine-tuning: instead of showing the model only good examples, it teaches the model to prefer one answer over another when both are plausible. That matters when quality is subjective, such as style, helpfulness, conciseness, or safety boundaries, because SFT tends to converge on an average behavior and cannot distinguish between merely correct and clearly better responses. The chapter positions preference tuning as the natural next step after SFT, while also warning that alignment work can introduce forgetting or safety regressions, so every new stage must be evaluated carefully.

The hands-on workflow centers on Direct Preference Optimization, which trains on prompt-level pairs labeled as chosen and rejected. The chapter shows how to reformat real human-graded support answers into the expected schema, then train a DPO model from the previous SFT checkpoint using TRL’s trainer. DPO is presented as practical because it removes the need for a separate reward model and reinforcement-learning loop, while still using a frozen reference model to constrain drift. Training is relatively lightweight, but the chapter emphasizes that good preference data, sensible hyperparameters, and enough GPU memory are essential for stable results.

Evaluation shows the key lesson: DPO does not necessarily improve objective correctness on factual help-desk tasks, but it can improve the subjective qualities that Token-F1 cannot see. In the example run, the DPO model roughly matched the SFT model on held-out correctness while learning the preferred style more clearly in a separate conciseness test. The chapter then broadens the view by comparing DPO with GRPO, RLHF, and RFT, explaining that GRPO and RFT are best when rewards are verifiable, RLHF is powerful but expensive and complex, and DPO is the default choice for most enterprise preference tuning. The final guidance is to choose the simplest method that matches the task, measure the right axis, and always rerun safety checks before shipping.

The progressive adaptation pipeline: each chapter’s output is the next chapter’s starting checkpoint.

DPO training dashboard: loss falls while reward margin and held-out reward accuracy rise.

The DPO training mechanism: raise the chosen response, lower the rejected one, stay near the frozen reference.

Per-category Token-F1 for the base, SFT, and DPO models: DPO and SFT track each other on every category.

One GRPO training step: sample a group of G responses, score them, normalize within the group, update toward the above-average ones.

RLHF’s three stages and four models versus DPO’s single supervised step.

The RFT loop: a rule-based verifier scores each output, and the policy is reinforced toward verified-correct answers.

Choosing among DPO, GRPO, RLHF, and RFT, entered only after the SFT-data-quality gate.

• SFT teaches a model what to output, while preference optimization teaches it to distinguish good responses from better ones.
• DPO collapses RLHF’s three-model pipeline into a single supervised training step on preference pairs, eliminating the need for a separate reward model.
• Preference data can come from human-graded pairs, as in this chapter’s run, or be generated automatically by contrasting a stronger model’s outputs against a weaker one’s at near-zero annotation cost.
• DPO does not move objective correctness (Token-F1 was flat on the IT eval, 0.168 to 0.166); it earns its place on subjective dimensions, where a conciseness-tuned DPO model produced answers 21% shorter, which a blinded judge preferred 61% of the time, while correctness held.
• The beta parameter (default 0.1) controls how closely DPO stays to the reference policy; values below 0.05 risk mode collapse, and values above 0.3 produce minimal change.
• GRPO eliminates PPO’s critic network by normalizing rewards within a group of sampled responses and excels when verifiable reward functions are available.
• RLHF remains the most capable alignment technique, but DPO achieves 80 to 90% of its benefit at roughly one-tenth the cost.
• Reinforcement fine-tuning with verifiable rewards replaces human preferences with code-based reward functions and is most applicable to domains like code generation, math, and structured extraction.
• The DeepSeek-R1 cold-start pattern combines light SFT for output format with large-scale GRPO for reasoning incentive, producing emergent self-verification behaviors.
• Known DPO failure modes include length and verboseness bias and likelihood displacement, both of which require monitoring after deployment.
• Safety regression testing must follow every preference optimization step because optimizing for helpfulness can erode refusal boundaries.
• Most enterprises (roughly 95%) need only DPO or better training data; reserve GRPO, RFT, RLHF, and frontier SFT+RL for specialized requirements.

FAQ