7 Reasoning & Inference-Time Scaling
The chapter explains how reasoning models became a major leap in language model performance by combining long chain-of-thought generation with inference-time scaling, where extra computation at generation time can produce better answers. It frames reinforcement learning as moving from a minor post-training add-on to a central tool for shaping model behavior, especially as models grew capable enough to benefit from extensive reasoning before responding. The result is a new family of “thinking” models that often spend many tokens exploring a problem, then return stronger outputs on tasks like math, code, and science.
A central idea is Reinforcement Learning with Verifiable Rewards, which adapts the RLHF pipeline but replaces subjective reward modeling with simple verification functions that can judge correctness directly. In math, this may mean checking whether an extracted final answer matches the known solution; in code, unit tests can serve as the verifier. Because the reward is usually binary or otherwise objective, the model can be trained by repeatedly sampling multiple answers, reinforcing the successful ones, and revisiting the same data many times to discover behaviors correlated with correctness.
The chapter also surveys the evolution of reasoning training, from earlier work that encouraged chain-of-thought on verifiable tasks to the modern large-scale RL recipes used by models like o1 and DeepSeek R1. It highlights common implementation choices such as difficulty filtering, curriculum scheduling, relaxed clipping, removal of KL penalties, format and language-consistency rewards, and parallel test-time sampling. Looking forward, the field is still moving quickly, but the main lesson is that reinforcement learning now plays a foundational role in frontier model training, and future systems will likely refine rather than abandon these ideas.
RLVR in the form of an RL feedback loop. Instead of a reward model, a verification function is used.
Summary
- Reasoning models use reinforcement learning with verifiable rewards (RLVR) to dramatically improve performance on math, code, science, and other tasks where the correctness of an answer can be assessed. Unlike RLHF, which requires a learned reward model, RLVR uses verification functions – such as answer matching or unit tests – that return definitive correctness signals.
- The core training loop applies the policy gradient algorithms from chapter 6 in a remarkably simple way: sample multiple answers to questions, take gradient steps toward the correct ones, and repeat.
- Inference-time scaling – using more computation during generation to get better responses – is closely linked to reasoning training. RL-trained models learn to produce longer reasoning chains that are strongly correlated with improved performance, a meaningful shift from the superficial length bias seen in early RLHF.
- Common practices for reasoning training include difficulty filtering (training only on problems the model solves 20-80% of the time), removing or relaxing KL penalties to allow greater exploration, format and language consistency rewards, and progressive length scheduling to combat overthinking.
- The reasoning model landscape evolved rapidly from OpenAI’s o1 and DeepSeek R1 through dozens of open-weight models – including Qwen 3, Phi-4 Reasoning, Llama-Nemotron, and OLMo 3 Think – with the field converging on recipes that combine instruction tuning, RLHF, and large-scale RLVR in carefully sequenced stages. Substantial changes to these recipes are expected to continue.
FAQ
What is inference-time scaling in reasoning models?
Inference-time scaling is the practice of improving model performance by spending more computation during generation, such as producing longer reasoning chains or sampling multiple candidate answers and selecting the best one.How does RLVR differ from RLHF?
RLHF relies on a learned reward model to judge subjective qualities like helpfulness or clarity, while RLVR uses a verification function that gives a definite reward based on correctness, such as whether an answer matches a ground truth or passes tests.Why are reasoning models considered a major breakthrough for language models?
Reasoning models showed that scaling reinforcement learning on verifiable tasks like math, coding, and science can dramatically improve performance, especially when models are trained to think extensively before answering.What kinds of tasks are best suited for RLVR?
RLVR works especially well in domains where correctness can be checked automatically, such as mathematics with exact answers, code generation with unit tests, and precise instruction following with verification rules.Why is the \boxed{} format useful in RLVR?
The \boxed{} notation makes it easy to extract the final answer from a response using a simple regular expression, which helps automated verification determine whether the answer is correct.
Reinforcement Learning from Human Feedback ebook for free
