Build a DeepSeek Model (From Scratch)

Learn how to build the features that set DeepSeek apart from other top LLMs!

When DeepSeek started making waves in January 2025, it sounded too good to be true. How could a generative AI model get such incredible performance with such low training and operation costs? By creatively blending a variety of strategies and innovations like Mixture of Experts, Latent Attention, Multi-token Prediction, model distillation, and efficient parallelization, DeepSeek set a new standard for what’s possible in an open LLM. Now, in Build a DeepSeek Model (From Scratch) you can recreate a laptop-scale version of this cutting-edge model yourself!

In Build a DeepSeek Model (From Scratch) you will learn how to:

• Implement DeepSeek’s core architectural innovations, including Multi-Head Latent Attention and Mixture-of-Experts layers
• Build a production-ready training pipeline with Multi-Token Prediction and FP8 quantization for efficiency and speed
• Maximize hardware utilization with parallelism strategies like DualPipe
• Apply post-training methods such as supervised fine-tuning and reinforcement learning to unlock reasoning capabilities
• Compress and distill large models into smaller, deployable versions for real-world use

In Build a DeepSeek Model (From Scratch) you’ll build your own DeepSeek clone from the ground up. First, you’ll quickly review LLM fundamentals, with an eye to where DeepSeek’s innovations address the common problems and limitations of standard models. Then, you’ll learn everything you need to create your own DeepSeek-inspired model, including the innovations that put DeepSeek on the map: Multihead Latent Attention (MLA), Multi-Token Prediction (MTP), Mixture of Experts (MoE), model distillation, and reasoning.