7 Improving GRPO for reinforcement learning
This chapter revisits the GRPO reinforcement-learning setup from the previous chapter and shifts from a short proof-of-concept run to a longer, more realistic training analysis. It emphasizes that even when the code is technically correct, training can still become unstable over time, so careful monitoring is essential. The discussion is grounded in actual experiments, but the results are treated as illustrative rather than definitive because RL runs can vary noticeably from one seed to another.
A major theme is learning how to interpret training behavior through metrics beyond raw reward and accuracy. The chapter explains how to track loss, average reward, response length, and external evaluation accuracy, then extends that toolkit with advantage statistics and entropy. Advantage mean is mostly a sanity check because it should stay near zero by construction, while advantage standard deviation reveals whether there is still a usable learning signal. Entropy adds another useful diagnostic: low values can indicate collapse or overconfidence, moderate values suggest healthy exploration, and sudden changes can warn of instability.
The chapter then improves the basic GRPO algorithm itself by adding clipped policy ratios and, separately, a KL regularization term. Clipping limits how far the policy can move in one update, which helps prevent large destructive jumps and makes longer training runs more stable. The KL term constrains the model relative to a reference policy, but the chapter also shows that it can sometimes contribute to collapse, especially when rewards vanish and the KL penalty becomes the main gradient signal. Finally, the chapter introduces an auxiliary format reward based on
A mental model of the topics covered in this book. This chapter provides a deeper coverage of the GRPO algorithm for reinforcement learning with verifiable rewards.
A chapter overview showing the different topics being covered in this chapter.
Output from a GRPO training run using GRPO in a terminal with several training statistics, such as the loss, average reward, tokens/sec throughput, and average response length.
The four metrics tracked during the GRPO training run (loss, average reward, average response length, and evaluation accuracy). The orange centerline represents a moving average over the last 25% of values, which helps reveal overall trends in the otherwise noisy training signals. The evaluation accuracy is shown as a bar plot since it is computed only every 50 steps rather than at each step.
After analyzing basic GRPO training metrics, we now add more advanced metrics to analyze the training run.
GRPO overview figure from chapter 6. The advantages are shown in step 3.
Log-probability (logprob) computation of a single token ("this") in the LLM's generated answer. The LLM returns the logits of the token, which are then converted to softmax probability values via torch.softmax() or logprob values via torch.log_softmax().
The entropy term is calculated by multiplying the token probabilities with the token logprobs.
Visualizing advantage statistics and entropy tracked during the GRPO training run (next to the average reward, which we tracked previously).
After plotting basic and advanced GRPO training metrics, we now modify the GRPO algorithm and add clipped policy ratios.
GRPO overview figure from chapter 6. We now use the sequence logprobs from step 4 to compute policy ratios and clipped policy ratios.
Calculating the policy ratio (ratio) and clipped policy ratio (clipped ratio) added to the GRPO from "new" and "old" logprobs.
Selected metrics from a GRPO training run using clipped policy ratios.
Implementing a KL loss term, which is a part of the original GRPO algorithm.
Overview of the GRPO algorithm with the KL loss term calculation added to the right.
Selected metrics from a GRPO training run after adding a KL loss term. Here, loss (in the upper left) refers to the total GRPO loss, that is, the sum of the policy-gradient loss and the KL loss term.
Implementing a format reward that encourages the model to generate <think>...</think> tokens
Using the previous correctness reward (left) and a correctness plus format reward (right).
Basic metrics from a GRPO training run with a format reward.
Additional metrics from a GRPO training run with a format reward.
This last section in this chapter outlines some additional GRPO modifications that emerged in recent months.
Summary
- Training reasoning models with GRPO can become unstable over longer runs, even when the implementation is correct and rewards initially improve.
- Interpreting GRPO training requires tracking multiple metrics jointly (average rewards, response length, evaluation accuracy, advantage statistics, and entropy)
- Basic metrics such as loss mainly serve as sanity checks in GRPO and should not be over-interpreted in isolation.
- Advantage statistics provide useful diagnostics: the mean should remain near zero by design, while the standard deviation reflects the strength and stability of the learning signal.
- Entropy measures how uncertain the model is during generation. Very low entropy can signal collapse, and very large entropy can indicate unstable updates and randomness in the model responses.
- Clipped policy ratios limit how much the policy can change between updates and can substantially improve training stability over longer runs.
- Adding a KL divergence term constrains long-term drift from a reference model but can destabilize training when rewards collapse.
- For math reasoning tasks, several recent systems report better stability and performance by omitting the KL term altogether.
- Auxiliary format rewards can improve the response structure, such as encouraging the use of <think> and </think> tokens.
- Beyond the original GRPO algorithm, many recent extensions modify advantage normalization, importance sampling, clipping strategies, and KL handling to improve stability and efficiency.
Build a Reasoning Model (From Scratch) ebook for free
