3 Training Overview

This chapter gives a broad introduction to reinforcement learning from human feedback by first grounding the reader in the basic reinforcement learning setup. It explains how a policy selects actions from a state, how the environment responds through transitions, and how reward is accumulated across a trajectory. Simple examples like a thermostat and CartPole are used to build intuition for the trial-and-error loop in which an agent improves its behavior over time by seeking higher return.

The chapter then shows how RLHF adapts that standard RL framework to language models. Instead of learning from an environment’s reward function and multi-step state transitions, RLHF usually starts from prompts, treats the completion as the action, and assigns reward at the level of the full response. A learned reward model stands in for the environment reward, and because the model is being fine-tuned from a pretrained base model rather than trained from scratch, regularization becomes essential. In particular, a KL penalty is used to keep the optimized policy close to the reference policy while still improving toward human preferences.

Finally, the chapter surveys the main optimization tools and canonical training recipes that appear throughout modern post-training. It introduces instruction fine-tuning, reward modeling, rejection sampling, policy gradients, and direct alignment methods as the core techniques used in RLHF pipelines. It also traces how these pieces were combined in influential systems such as InstructGPT, then expanded into more complex multi-stage recipes like Tülu 3 and DeepSeek R1, illustrating the field’s shift toward larger, more iterative, and more specialized training workflows.

Standard RL loop

Each term in the trajectory distribution (equation 3.1) mapped to the thermostat RL example.

CartPole environment showing state variables (\(x\), \(\dot{x}\), \(\theta\), \(\dot{\theta}\)) + and actions (\(\pm F\)).

Standard RLHF loop

A rendition of the early, three stage RLHF process with SFT, a reward model, and then optimization.

A rendition of modern post-training with many rounds.

A summary of the Tülu 3 recipe with target skills and multi-step training recipe. Lambert et al. 2024, License CC-BY.

• In reinforcement learning, an agent takes actions sampled from a policy given the current state to maximize cumulative reward – RLHF adapts this framework for language models.
• RLHF builds on the standard RL setup but with key differences: a learned reward model replaces the environment reward function, there are no state transitions (prompts are sampled from a dataset), and rewards are assigned at the response level rather than per-step.
• To prevent the model from drifting too far from its starting point, RLHF adds a KL divergence penalty between the current policy and a reference policy – colloquially, this optimization distance can be viewed as a budget to spend in more complex training setups.
• This chapter also previews the main optimization tools used in post-training, including reward modeling, instruction fine-tuning, rejection sampling, policy gradients, and direct alignment algorithms.
• The InstructGPT-era recipe established the canonical three steps: instruction tuning on  10K examples, training a reward model on  100K preference pairs, and optimizing the model with RL on  100K prompts.
• Modern recipes like Tülu 3 involve many more stages and orders of magnitude more data, combining instruction tuning, on-policy preference data, and reinforcement learning with verifiable rewards (RLVR).

FAQ