8 Distilling reasoning models for efficient reasoning

Distillation is presented as a practical training-time route to better reasoning performance: instead of relying only on inference-time scaling or reinforcement learning, a smaller student model learns from reasoning traces and final answers produced by a stronger teacher. The chapter contrasts hard distillation, where the student imitates teacher-generated text, with soft distillation, where it matches the teacher’s probability distribution; while soft distillation can carry richer information, hard distillation is far more practical for most LLM workflows because it only needs generated text and not logits.

The implementation focuses on a DeepSeek-R1-style hard distillation pipeline built around math problems. A teacher model generates reasoning traces and answers for a 12,000-example MATH subset, the data is formatted with optional thinking tags, tokenized with a reasoning-aware tokenizer, filtered for excessive length, and split into training and validation sets. The chapter emphasizes careful preprocessing because distillation reuses the same examples across epochs, making it worthwhile to cache the formatted sequences and keep only answer-token positions for the loss.

Training uses a pre-trained Qwen3 0.6B base model and a standard cross-entropy objective applied only to the teacher answer tokens, not the prompt. A small training run demonstrates that validation loss falls as the student learns the teacher’s outputs, and larger runs show meaningful gains on MATH-500, especially when teacher and student share the same model family and formatting conventions. The chapter closes by placing distillation in the broader reasoning-model landscape, where it remains a key method for transferring strong reasoning behavior into smaller, cheaper models that are easier to deploy locally or inside agentic systems.

A mental model of the topics covered in this book. This chapter focuses on distillation, where a smaller student model is trained on reasoning traces generated by a larger teacher model.

Hard distillation trains the student on teacher-generated tokens, soft distillation trains the student on the teacher's full output distribution.

The chapter has four main steps: (1) distillation introduction and overview; (2) dataset preparation (steps 2a-2c); (3) training via distillation (steps 3a-3c); and (4) evaluation.

Distillation setup used in this chapter. We use the 12,000 non-overlapping MATH training problems to obtain synthetic solutions from DeepSeek-R1 and later evaluate the distilled Qwen3 student on the separate MATH-500 test set.

In RLVR, the generated answer is compared against the ground-truth reference solution (top subpanel), whereas in distillation the student answer is compared against the teacher-generated solution (bottom subpanel).

Chapter overview with the current section highlighted. Here, we load the DeepSeek-R1-generated dataset from a JSON file before preparing it for training.

Bringing the loaded dataset into a format suitable for model training by understanding the tokenizer, tokenizing the examples, and filtering and splitting the dataset.

With the tokenizer step complete, we now move on to apply the formatting and tokenization steps to the whole dataset.

Example of the tokenization pipeline for one training sample. The math problem is rendered into the chat prompt format, the teacher reasoning trace and final answer are combined via format_distilled_answer, and both parts are concatenated into one token sequence.

After tokenization, we filter out long sequences and split the remaining examples into training and validation subsets.

With the dataset preparation complete, we begin the distillation training by loading the pre-trained Qwen3 base model.

Illustration of token and sequence log-probabilities. The log-probabilities of the correct next tokens are summed to obtain the sequence log-probability, which is the basis for the cross-entropy loss used later in this chapter.

Input for the cross-entropy loss over the answer tokens. The model receives the prompt and answer shifted by one token as input, and the answer-token logits are compared against the reference answer tokens.

With dataset preparation and loss computation complete, we now turn to the training loop for distillation.

Distillation training loop. In each epoch, the training examples are shuffled, the student model computes a cross-entropy loss for each example, gradients are backpropagated, and the model weights are updated. The validation loss is reported in certain intervals to track progress.

After implementing the training loop, we evaluate the distilled model on the MATH-500 test set.

The training and validation loss of a 3-epoch distillation training run on DeepSeek-R1 reasoning traces.

The evaluation of the distilled model completes the technical content of this chapter.

• Distillation trains a smaller student LLM on outputs produced by a larger teacher LLM.
• Hard distillation is usually more practical than soft distillation because teacher logits are often unavailable and teacher text outputs are much cheaper to store and reuse.
• We used DeepSeek-R1 as the teacher model and Qwen3 0.6B as the student model
• The distillation dataset was built from the 12,000 MATH training problems that do not overlap with MATH-500.
• Each training sample combines a rendered prompt with the teacher reasoning trace and final answer, optionally separated via <think>...</think> tags.
• For efficiency, we tokenize the dataset once, filter it by sequence length, and reuse the processed examples across multiple epochs.
• The training objective is answer-only cross-entropy, which is equivalent to the negative average log-probability of the correct next tokens.
• The distillation training loop is a standard supervised learning loop, which includes shuffling the training examples each epoch, computing the loss, backpropagating, updating the model weights, and tracking validation loss.
• The validation loss is the main signal to watch during training, and saved checkpoints can later be evaluated on MATH-500 with the verifier from chapter 3.
• The distillation approach in this chapter improves the Qwen3 0.6B base model from 15.2% accuracy on MATH-500 to 45.0% accuracy.

FAQ