Overview
8 Knowledge distillation: Making powerful models practical
After adding strong reasoning through reinforcement learning, the book turns to a practical problem: the most capable DeepSeek model is far too large and expensive to run on ordinary hardware. The chapter explains that knowledge distillation is the answer, allowing the behavior and reasoning skill of a huge teacher model to be compressed into much smaller student models that can run on a laptop, GPU, or even a phone. This makes advanced reasoning usable in real-world settings where cost, memory, latency, and connectivity all matter.
The core idea is the teacher-student paradigm, where the student learns not just the correct answer but also the teacher’s softer probability distribution over alternatives. Those softened outputs reveal “dark knowledge,” meaning the subtle relationships among classes that hard labels hide. Temperature scaling is the key mathematical tool: it spreads out the teacher’s probabilities so the student can see meaningful structure, but not so much that the signal becomes uniform and useless. The chapter also shows why the soft-target loss must be multiplied by temperature squared to keep gradients strong enough to matter, and how the final training objective combines hard-label cross-entropy with soft-target KL divergence.
The chapter then shifts from classical distillation to DeepSeek-R1’s own recipe for reasoning transfer. Instead of matching logits, DeepSeek generates high-quality chain-of-thought traces, filters them with rejection sampling, and fine-tunes smaller open-source base models on about 800K curated examples. The results are striking: distilled models are dramatically smaller yet highly competitive, with the 1.5B version beating GPT-4o on mathematical reasoning and larger distilled models outperforming o1-mini across several benchmarks. The chapter closes by emphasizing both the promise and the limits of distillation: it is a highly effective way to deploy learned intelligence, but it cannot create capabilities beyond what the teacher already knows.
This chapter focuses on the highlighted component, Knowledge Distillation, the final piece of Stage 4’s post-training pipeline.
The deployment gap between DeepSeek-R1 (requiring a server cluster) and real-world targets where reasoning capabilities are needed, from data center GPUs down to mobile devices.
The dramatic reduction in memory and GPU requirements from the 671B teacher model down to the smallest 1.5B distilled model, which runs on a phone.
The teacher-student paradigm in knowledge distillation. The frozen teacher shares its soft probability distributions with the student, transferring richer information than hard labels alone.
Hard labels provide approximately 1 bit of information per sample (correct or incorrect), while the teacher’s soft labels reveal rich inter-class similarity structure. The dark knowledge.
Learning from a recipe book (hard labels) gives you the correct answer but no context, while learning from a master chef (soft labels) reveals the relationships between alternatives. The dark knowledge that enables deeper understanding.
Standard softmax at T=1 for logits [5.0, 3.0, 1.0, -1.0]. The distribution is heavily peaked at “cat” (86.5%), hiding the dark knowledge in near-zero probabilities.
The same logits [5.0, 3.0, 1.0, -1.0] processed through softmax at three temperatures. At T=1, the distribution is too peaked, at T=3 the dark knowledge emerges clearly, and at T=10 the signal is lost in near-uniformity.
At T=3, the teacher’s soft targets reveal the inter-class similarity structure it has learned. Cats are somewhat dog-like (26.8%), slightly fox-like (13.8%), and very unlike birds (7.1%). This similarity structure IS the dark knowledge.
The complete step-by-step computation of soft targets at T=3: logits are divided by T, exponentiated, and normalized to produce the final probability distribution.
The temperature Goldilocks zone. At T=1, the dark knowledge is hidden, between T=2 and T=5, it is optimally revealed, and beyond T=10, the useful signal is lost. Smaller student models typically benefit from lower temperatures within this range.
Attempt #1: Training the student from scratch on hard labels. Each sample provides approximately 1 bit of useful information, and the student has no way to learn inter-class relationships.
Attempt #2: Using the teacher’s soft output probabilities at T=1 as training targets. The distribution is still too peaked. The dark knowledge remains hidden in near-zero values, providing only marginal improvement over hard labels.
The breakthrough: applying temperature T=3 to the softmax unlocks the dark knowledge. The teacher’s probability distribution now reveals rich inter-class relationships, dramatically improving the students’ learning.
The complete knowledge distillation loss function. Two parallel paths, hard-label cross-entropy (grounding) and soft-target KL divergence (knowledge transfer), combine into a single weighted objective.
The rejection sampling process for curating DeepSeek-R1’s distillation dataset. Multiple responses are generated per prompt, verified for correctness, filtered for quality, and only clean, correct answers are kept.
DeepSeek-R1’s complete training pipeline. Distillation (Stage 6) is the final stage, where open-source dense models are fine-tuned on 800K samples curated from the teacher via rejection sampling. Data generation costs only $10K versus $202K for RL training. (Cost figures are estimates derived from the H800-hour counts disclosed in the DeepSeek-R1 paper at published cloud GPU rates; exact internal costs may differ.)
The composition of DeepSeek-R1’s 800K distillation dataset by domain. Math dominates at 395K samples (49%), followed by Code at 211K (26%), with General, STEM, and Logic making up the remainder. (Estimated split; the R1 paper itself reports only the ~600K reasoning / ~200K non-reasoning totals.)
Two paradigms of knowledge distillation. Classical KD (left) transfers knowledge through logit-matching with KL divergence and temperature scaling. DeepSeek’s approach (right) transfers knowledge through supervised fine-tuning on the teacher’s generated chain-of-thought text.
DeepSeek’s six distilled models span two architecture families: Qwen2.5 (1.5B, 7B, 14B, 32B) and Llama-3 (8B, 70B). The 1.5B and 7B Qwen models use math-specialized base models, while larger models use general-purpose bases.
The toy-sized teacher model (4 layers, 128-dim hidden, ~26K parameters) and student model (2 layers, 32-dim hidden, ~8.8K parameters) used in our implementation. The student is 3x smaller but maintains the same input and output dimensions.
One training step in knowledge distillation. The frozen teacher and the trainable student both process the input. Two loss terms are computed and combined, and only the student’s weights are updated.
Step-by-step computation of the combined distillation loss with actual numbers, showing how teacher and student logits flow through softmax at T=3, KL divergence, T² scaling, and the final weighted combination.
AIME 2024 benchmark results comparing DeepSeek’s distilled models against frontier competitors. The 1.5B distilled model outperforms GPT-4o (28.9% vs 9.3%), and the 32B model outperforms o1-mini (72.6% vs 63.6%).
The David vs Goliath moment: a 1.5B-parameter distilled model outperforms both GPT-4o and Claude-3.5-Sonnet—frontier models whose exact sizes are undisclosed but far larger by any estimate—on mathematical reasoning benchmarks.
Distillation vs direct RL on the same 32B Qwen2.5 architecture. The distilled model (72.6% AIME) outperforms the RL-trained model (47.0%) by 25.6 percentage points across all benchmarks.
Performance (AIME 2024) scales smoothly with model size for distilled models. Qwen-Math bases consistently outperform Llama bases at similar sizes, showing that the base model’s pre-existing capabilities interact with distillation quality.
The capacity gap in action. The 1.5B distilled model excels at mathematical reasoning (83.9% MATH) but struggles with coding (16.9% LiveCodeBench), indicating that its limited capacity cannot absorb all domains equally.
The knowledge frontier. Distillation efficiently transfers existing knowledge (chain-of-thought patterns, factual knowledge, mathematical strategies), but novel reasoning capabilities and frontier intelligence still require reinforcement learning and more powerful base models.
Summary
- Knowledge distillation transfers a large teacher model’s capabilities to a smaller student model by training the student on the teacher’s soft probability distributions rather than hard one-hot labels.
- Temperature-scaled softmax with T > 1 reveals dark knowledge -the relative probabilities of incorrect classes that encode inter-class similarity structure invisible in hard labels.
- The classical distillation loss combines two terms: a hard-label cross-entropy loss (grounding) and a soft-target KL divergence loss scaled by T² (knowledge transfer), weighted by coefficients α and β.
- DeepSeek-R1’s distillation uses a fundamentally different approach: supervised fine-tuning on 800K chain-of-thought reasoning traces generated by the teacher via rejection sampling, with no temperature scaling or KL divergence.
- DeepSeek’s 800K training dataset contains approximately 395K math, 211K code, 10K STEM, 10K logic, and 178K general samples -curated through rejection sampling that retains only verified-correct, well-formatted responses.
- The distilled DeepSeek-R1-Qwen-1.5B model outperforms GPT-4o on AIME 2024 (28.9% vs 9.3%), demonstrating that reasoning capabilities can be compressed by a factor of approximately 450x in parameter count.
- Distillation dramatically outperforms direct reinforcement learning on the same architecture: the distilled 32B model achieves 72.6% on AIME versus 47.0% for the RL-only variant, a gap of 25.6 percentage points.
- The capacity gap limits what very small students can absorb -a 1.5B model excels at math (83.9% MATH) but struggles with coding (16.9% LiveCodeBench).
- Distillation transfers existing knowledge efficiently but cannot create new capabilities -advancing beyond the teacher’s frontier still requires more powerful base models and larger-scale reinforcement learning.
Build a DeepSeek Model (From Scratch) ebook for free
