7 Reinforcement learning from human feedback

In this chapter, the central idea is that evaluation and alignment are deeply connected: many evaluation metrics already produce scalar scores that can be used as training signals, but the real challenge is deciding when those scores are trustworthy enough to guide optimization. The chapter frames this as a shift from measuring model quality to improving model behavior, using a financial research assistant as a recurring example. It also distinguishes between verifiable tasks, where a programmatic checker can directly confirm correctness, and open-ended tasks, where human judgment or a learned proxy is needed because there is no single ground truth.

The chapter then explains the standard post-training path for foundation models: pretraining gives broad knowledge, supervised fine-tuning teaches instruction following, and reinforcement learning pushes the model toward better-than-demonstrated behavior. Reinforcement learning is presented in language-model terms as a policy that generates responses, a reward that scores them, and an update step that shifts the model toward higher-reward outputs. Because language models do not receive rewards from an environment the way games or simulators do, the hard part becomes designing the reward itself, and the chapter shows why simple metric-based rewards often fail through reward hacking and Goodhart’s law.

To solve reward specification, the chapter traces the development of RLHF: learning a reward model from human preferences, scaling that recipe into InstructGPT, and then extending it with safer and richer reward designs. It shows how pairwise comparisons can train a reward model, why PPO is paired with a KL penalty to prevent drift, and how later methods such as best-of-N sampling, rejection sampling, RLAIF, process supervision, verifiable rewards, and evolving composite rubrics address weaknesses in classic RLHF. The broader lesson is that alignment-by-training is not one technique but a family of reward-engineering strategies, chosen according to whether a task is verifiable, open-ended, or a mix of both.

Training of an AI model is typically split into different stages - pretraining and post-training. Mid-training might be split into its own stage operationally.

The reinforcement-learning (RL) loop. The agent learns to make decisions by acting in an environment, observing a state and reward, and updating its policy to maximize cumulative reward.

A trajectory is the steps that the agent has taken to complete a task to produce a final response. Agents typically go through loops of reason -> act -> result to complete tasks.

The reinforcement learning training loop. A prompt enters the policy, which samples a response. The response is scored by a reward function. The scalar reward drives a gradient update that adjusts the policy's parameters. The loop repeats over many prompts and sampled responses

The three concurrent processes in Christiano et al.'s 2017 system. The policy generates trajectory segments that feed a preference-labeling queue. Human annotators compare pairs and return binary judgments, which accumulate in the preference dataset. The reward predictor is trained on the dataset, and its scores drive the policy-gradient updates. Arrows show the data flow; all three loops run continuously rather than in sequence.

An example of a preference labeling interface for human annotators. This is an example of label text data, not the interface used in Christiano et al.

Mapping Christiano et al.'s 2017 RL from human preferences to AI post-training.

The InstructGPT three-stage pipeline. Stage 1 fine-tunes the pretrained model on labeler-written demonstrations. Stage 2 trains a reward model on labeler-ranked comparisons of model outputs. Stage 3 optimizes the SFT model with reinforcement learning using PPO, with the reward model's score (penalized by KL divergence from the SFT model) as the reward signal. The SFT model from stage 1 plays two roles: it initializes the RL policy and is the reference distribution in the KL penalty. Source: Figure 2 of Ouyang et al. 2022.

MATH accuracy as a function of the number of reranked samples N, for three rerankers: PRM, ORM, and majority voting. The PRM curve sits above the ORM curve across the full range, and both sit above majority voting.

A composite reward with evolving rewards. It combines rule-based rewards with positive rubrics that the annotator marks as high-value and negative rubrics that penalize newly observed failure modes and reward-hacking patterns during training. The total reward is a weighted sum, and the negative-rubric set grows over training.

• Evaluation metrics and training rewards are the same kind of scalar signal in different roles. Alignment is multi-dimensional: a single labeler preference collapses judgments across the principled, policy, and personality pillars into one bit. This is why RLHF can improve all three at once but cannot be steered along any one axis without additional structure.
• Pretraining gives a model knowledge but not steerability. SFT teaches it to imitate demonstrations, but it cannot penalize behaviors it has never seen and cannot represent preferences between plausible completions. Reinforcement learning closes both gaps by training against a reward signal rather than a fixed target, at the cost of a credit-assignment problem that gets worse as responses get longer.
• Fixed-metric rewards fail under optimization because reinforcement learning exploits any gap between the proxy and the underlying objective. This structural form of Goodhart's law in AI is known as reward hacking.
• Deep Reinforcement Learning from Human Preferences (Christiano et al., 2017) showed that human preferences could replace hand-crafted rewards. The recipe ran three concurrent processes (rollout, labeling, reward-model training) and used ensemble disagreement to select informative pairs, demonstrating that low thousands of labels were enough to learn novel behaviors like a simulated robot backflip. This showed that preference learning can shape policies toward behaviors no engineer could specify by hand.
• Training language models to follow instructions with human feedback (InstructGPT; Ouyang et al., 2022) scaled the recipe to a production pipeline: SFT on labeler demonstrations, reward-model training on labeler rankings, and PPO with a KL penalty against the SFT reference. On held-out API prompts, labelers preferred 175B InstructGPT to 175B GPT-3 85% of the time; the 1.3B PPO-ptx model was also preferred to 175B GPT-3 despite being much smaller.
• The InstructGPT recipe is the conceptual template most modern post-training pipelines still adapt, with components swapped in and out. New optimizers such as DPO/GRPO replace PPO; RLAIF replaces human labelers with an LLM judge in stage 2; best-of-N and rejection sampling reuse a trained reward model at inference or in iterated SFT loops as cheaper substitutes for full PPO.
• RLHF's failure modes show up as length bias, sycophancy, surface-pattern hijacking, label-pool bias, and credit assignment that gets worse with longer responses. A KL penalty helps limit drift during PPO but does not solve reward hacking by itself. Annotation quality and labeler representativeness set the ceiling of what any RLHF program can achieve.
• Reward design has moved from a single learned reward model to a richer set of signals. Practitioners have to decide which components of their task have ground truth, which need preferences, and how the reward composition changes as the model improves. Process supervision catches reasoning errors that outcome supervision cannot see. Verifiable rewards skip reward modeling when programmatic ground truth exists. Reference-based statistical rewards such as BLEUBERI can work when a strong base model, constrained updates, a curated prompt distribution, and reliable reference answers limit metric exploitation. Composite and evolving recipes mix rule-based verifiers, learned reward models, and dynamic positive/negative rubrics that change across training.

FAQ