7 Knowledge distillation: Capturing frontier model intelligence

This chapter shows how knowledge distillation can turn a strong teacher model into a smaller, cheaper student that preserves much of the teacher’s behavior. In practice, the process looks like supervised fine-tuning with a different data source: prompts are sent to a teacher, the responses are collected, filtered for quality, and then used to train a student on your own infrastructure. The appeal is not just performance, but the combination of lower cost, better privacy, lower latency, and greater operational independence. The chapter frames this as a black-box pipeline that works whether the teacher is a local model or a frontier API.

The hands-on workflow emphasizes that most of the success comes from the data pipeline rather than the training code. Good prompt design matters, especially diversity across normal cases, edge cases, difficult cases, and even adversarial prompts. Teacher outputs must then be screened for truncation, repetition, refusals on benign inputs, contamination with test examples, and other low-quality patterns before training begins. Once the verified data is ready, the student is trained with the same LoRA and SFT machinery used earlier in the book, and evaluation compares base, teacher, and student side by side. The key metric is retention: how close the student comes to the teacher’s score, with category-level analysis helping identify where the student keeps up and where it falls behind.

The chapter also explains when distillation is the right tool and when it is not. It is most attractive for stable tasks with enough query volume to justify the upfront work, especially when privacy, latency, or cost pressures make direct API use undesirable. However, distillation is lossy: it compresses behavior rather than creating new capability, and safety alignment does not reliably transfer unless refusal examples are explicitly included. The chapter closes by comparing forward and reverse KL as optimization choices, noting that standard forward-KL-style training is usually enough for enterprise use, while more specialized divergence methods matter only in narrower cases. Overall, the message is that distillation is a practical way to capture frontier-model behavior, but only if the data, evaluation, and safety checks are handled carefully.

Black-box distillation pipeline

Teacher data quality filtering and category distribution after the 159/40 split.

Student distillation training dashboard: a healthy run, with loss falling to 1.2, validation loss tracking it, and token accuracy around 69%.

Three-way Token-F1 comparison across categories on the held-out test split: the distilled student retains about 98% of the teacher’s score.

White-box vs. black-box distillation: black-box trains on the teacher's text outputs with any API; white-box trains on the teacher token probabilities and needs access to the teacher weights.

DeepSeek-R1 retention by student size: MATH-500 holds above 86% at every size, while AIME 2024 falls to 36% at 1.5B.

• Distillation is supervised fine-tuning where the training data comes from a stronger teacher model rather than from human annotators; the student inherits the teacher's behavioral patterns at a fraction of the inference cost.
• Distillation is a capital investment: spending $1,000 to $2,000 upfront on teacher data generation and student training saves $1,000+ per month in ongoing API costs, with break-even typically at one to three months.
• Black-box distillation (training on teacher text outputs) is the practical default for enterprise work because it works with any teacher model, including proprietary APIs.
• The commonly cited 70 to 90% retention range is heavily task-dependent: well-matched tasks can exceed 95%, while hard reasoning tasks with small students can drop well below 70%.
• Data quality drives student quality: filtering degenerate teacher outputs (terse answers, repetition) is a cheap safeguard worth keeping even when few examples are caught.
• Distillation reproduces the teacher's behavior cheaply. Still, it copies the teacher's gaps too: in our run, the student matched the teacher's house format on every held-out test prompt and reached about 98% of its Token-F1, yet it lost harmful-request refusals the base model had. Safety alignment does not carry over via distillation, so each student model must be independently tested for safety regressions before deployment.

FAQ