7 Improving GRPO for reinforcement learning

This chapter takes the basic GRPO-based reinforcement learning setup from the previous chapter and examines what happens when training is run for longer. It emphasizes that reward and accuracy alone are not enough to understand training behavior: loss, response length, evaluation accuracy, advantage statistics, entropy, and policy ratios all provide complementary signals. The chapter shows that early improvements can be followed by instability, such as declining benchmark accuracy, growing or shrinking response lengths, reward saturation, or loss spikes, even when the implementation is technically correct.

A major focus is stabilizing GRPO. The chapter introduces advantage tracking, where the average advantage should remain near zero and the standard deviation indicates whether a useful learning signal remains. It also adds entropy tracking to monitor whether the model is becoming too deterministic, too random, or still exploring productively. To reduce overly large updates, the chapter implements clipped policy ratios, which compare old and new sequence log-probabilities and limit how much a rollout can influence an update. This modification leads to more stable training in the demonstrated run. The chapter also revisits the KL loss term, explaining that it constrains the trained model to remain close to a reference model, but also showing that a naive KL implementation can destabilize training by encouraging long outputs or dominating the gradient when rewards collapse.

The chapter then extends the reward design by adding an explicit formatting reward, encouraging the model to produce reasoning inside <think> and </think> tags. It explains how tokenizers must support these tags, why a model should ideally be pre-trained or instruction-tuned on such tokens before RL training, and how a format reward can be combined with correctness rewards. The experiments show that auxiliary rewards can help shape output structure but may also be exploited if they are too strong or granted independently of correctness. The chapter closes by highlighting that GRPO and RLVR remain active research areas, with many recent modifications aimed at improving stability, sampling efficiency, normalization, clipping, KL handling, and reward aggregation.

A mental model of the topics covered in this book. This chapter provides a deeper coverage of the GRPO algorithm for reinforcement learning with verifiable rewards.

A chapter overview showing the different topics being covered in this chapter.

Output from a GRPO training run using GRPO in a terminal with several training statistics, such as the loss, average reward, tokens/sec throughput, and average response length.

The four metrics tracked during the GRPO training run (loss, average reward, average response length, and evaluation accuracy). The orange centerline represents a moving average over the last 25% of values, which helps reveal overall trends in the otherwise noisy training signals. The evaluation accuracy is shown as a bar plot since it is computed only every 50 steps rather than at each step.

After analyzing basic GRPO training metrics, we now add more advanced metrics to analyze the training run.

GRPO overview figure from chapter 6. The advantages are shown in step 3.

Log-probability (logprob) computation of a single token ("this") in the LLM's generated answer. The LLM returns the logits of the token, which are then converted to softmax probability values via torch.softmax() or logprob values via torch.log_softmax().

The entropy term is calculated by multiplying the token probabilities with the token logprobs.

Visualizing advantage statistics and entropy tracked during the GRPO training run (next to the average reward, which we tracked previously).

After plotting basic and advanced GRPO training metrics, we now modify the GRPO algorithm and add clipped policy ratios.

GRPO overview figure from chapter 6. We now use the sequence logprobs from step 4 to compute policy ratios and clipped policy ratios.

Calculating the policy ratio (ratio) and clipped policy ratio (clipped ratio) added to the GRPO from "new" and "old" logprobs.

Selected metrics from a GRPO training run using clipped policy ratios.

Implementing a KL loss term, which is a part of the original GRPO algorithm.

Overview of the GRPO algorithm with the KL loss term calculation added to the right.

Selected metrics from a GRPO training run after adding a KL loss term. ​​Here, loss (in the upper left) refers to the total GRPO loss, that is, the sum of the policy-gradient loss and the KL loss term.

Implementing a format reward that encourages the model to generate <think>...</think> tokens

Using the previous correctness reward (left) and a correctness plus format reward (right).

Basic metrics from a GRPO training run with a format reward.

Additional metrics from a GRPO training run with a format reward.

This last section in this chapter outlines some additional GRPO modifications that emerged in recent months.

• Training reasoning models with GRPO can become unstable over longer runs, even when the implementation is correct and rewards initially improve.
• Interpreting GRPO training requires tracking multiple metrics jointly (average rewards, response length, evaluation accuracy, advantage statistics, and entropy)
• Basic metrics such as loss mainly serve as sanity checks in GRPO and should not be over-interpreted in isolation.
• Advantage statistics provide useful diagnostics: the mean should remain near zero by design, while the standard deviation reflects the strength and stability of the learning signal.
• Entropy measures how uncertain the model is during generation. Very low entropy can signal collapse, and very large entropy can indicate unstable updates and randomness in the model responses.
• Clipped policy ratios limit how much the policy can change between updates and can substantially improve training stability over longer runs.
• Adding a KL divergence term constrains long-term drift from a reference model but can destabilize training when rewards collapse.
• For math reasoning tasks, several recent systems report better stability and performance by omitting the KL term altogether.
• Auxiliary format rewards can improve the response structure, such as encouraging the use of <think> and </think> tokens.
• Beyond the original GRPO algorithm, many recent extensions modify advantage normalization, importance sampling, clipping strategies, and KL handling to improve stability and efficiency.

FAQ