7 Reinforcement learning: From policy gradients to GRPO
This chapter shifts from building a strong base language model to improving its behavior after pretraining. It explains how reinforcement learning can turn text prediction into deliberate problem solving by rewarding complete responses instead of supervised next-token targets, using language generation as a sequential decision process with prompts as states, sampled tokens as actions, and final scores as rewards. The chapter’s core motivation is that reasoning tasks often have delayed, sparse feedback, so the model must learn which intermediate behaviors help it reach a correct final answer.
The chapter then develops the main policy-gradient ideas used to train language models from rewards. It shows why sampling is necessary for exploration, how raw rewards are noisy, and why advantages relative to a baseline are more stable than using rewards alone. PPO is presented as a practical solution that keeps policy updates near the data-generating policy with clipped ratios, while also relying on a value model for a learned baseline and a frozen reference model for KL control. In contrast, GRPO removes the value model and replaces it with group-relative normalization: multiple completions are sampled for the same prompt, scored, and compared against one another so that better completions get positive advantages and worse ones get negative ones.
The chapter closes by connecting these methods to the DeepSeek-R1 family and to a minimal PyTorch implementation. It explains how verifiable rewards from deterministic checkers can drive reasoning on math and code tasks, why cold-start supervised examples and later mixed-data fine-tuning are still needed for readability and general assistant behavior, and how rejection sampling turns strong RL outputs into supervised data for distillation. The code walkthrough makes the algorithm concrete with a verifier, group-relative advantage calculation, a clipped GRPO loss, and a one-step training skeleton, while also emphasizing real-world concerns such as masking, parsing, KL monitoring, reward hacking, and the distinction between algorithmic ideas and production-scale training systems.
Our four-stage journey to build the DeepSeek model. This chapter moves into post-training, where reinforcement learning, GRPO, verifiable rewards, and reference-policy control become central.
The agent-environment interface. The cleaning robot observes the room, chooses an action, receives a reward, and then repeats the cycle from the updated room state.
Mapping reinforcement learning concepts to LLM generation. The growing context is the state; each sampled next token is an action, and the final response receives a reward.
Sparse reward timing in LLM reinforcement learning. Tokens are generated one at a time, but the task reward is often assigned only after the full response is complete.
A probability distribution over the next token. During RL training, the model samples from this distribution rather than always selecting the highest-probability token.
PPO-style language-model training uses a trainable policy, a reward signal, a learned value baseline, and a frozen reference policy for KL control.
GRPO group sampling with reference control. Multiple completions are generated for the same prompt, scored, normalized into advantages, and updated under a KL constraint.
Group-relative advantage calculation. For binary rewards [1, 1, 0, 0], the group mean is 0.5, so correct completions receive positive advantages and incorrect completions receive negative advantages.
From PPO to GRPO to GRPO with verifiable rewards. GRPO removes the value model; RLVR replaces a learned reward model only for objectively checkable tasks.
RLVR can replace learned reward models for objectively verifiable tasks such as math and code; open-ended assistant behavior still requires preference signals.
The aha moment example from DeepSeek-R1 training. The <think> tag marks the model's reasoning region; the notable behavior is that the model notices a flawed path and changes strategy.
Accuracy and response length of DeepSeek-R1-Zero during RL training. Source: Guo et al., DeepSeek-R1, Figure 1.
The DeepSeek-R1 model lineup. R1-Zero demonstrates emergent reasoning; R1 improves readability and robustness; Distill transfers R1-generated reasoning data into smaller students.
DeepSeek-R1 training pipeline, adapted from Guo et al., DeepSeek-R1, Figure 2. The flow from R1-Zero into R1 represents filtered training data, not direct parameter transfer.
Implementation workflow for the chapter code. Prompts are sampled, completions are verified, rewards are normalized, and the policy is updated with a clipped GRPO objective plus KL control.
Summary
- LLM reinforcement learning treats the growing context as the state and each sampled token as an action.
- Policy-gradient methods update the probability of sampled actions in proportion to reward or advantage.
- PPO stabilizes policy updates with clipping and commonly uses a learned value model as a baseline.
- The value function Vϕ(s) is produced by a learned value model or value head trained from rollout returns.
- GRPO removes the learned value model by normalizing rewards within groups of completions sampled for the same prompt.
- GRPO is cheaper than PPO for the same total number of sampled completions because it eliminates the value-model path, not because it samples fewer responses.
- RLVR replaces learned reward models only when a task has objectively verifiable answers, such as many math and code tasks.
- DeepSeek-R1-Zero demonstrates emergent reasoning from GRPO plus verifiable rewards; DeepSeek-R1 adds staged SFT and RL to make the behavior more usable.
- The flow from R1-Zero into R1 is a data flow: filtered outputs become training data, not direct parameter transfer.
- The chapter code provides concrete hooks for verifier rewards, group-relative advantages, clipped GRPO loss, and a training-step skeleton.
Build a DeepSeek Model (From Scratch) ebook for free
