6 The DeepSeek training pipeline: Building a foundation model

The chapter brings together the individual DeepSeek ideas developed earlier into a working “mini-DeepSeek V3” system, showing how to move from theory to a complete training pipeline. It begins by preparing the TinyStories dataset, choosing a compact English corpus that is manageable on a consumer GPU, then tokenizing it with the GPT-2 tokenizer from tiktoken and saving the results efficiently as memory-mapped binary files. This setup also includes saving metadata like vocabulary size so the training script can initialize the model correctly.

Next, it builds the model itself in PyTorch by combining rotary positional encoding, decoupled MLA-style attention with a KV cache, mixture-of-experts feed-forward layers with routed and shared experts, and a causal multi-token prediction module. These parts are wrapped into Transformer blocks and then into a full MiniDeepSeek model that behaves differently for training and inference. During inference, it uses cached keys and values to generate text efficiently one token at a time; during training, it processes full sequences, computes the main next-token loss, and adds auxiliary MTP losses to strengthen learning signals.

The chapter then explains the full training script: loading tokenized data from disk, sampling batches, using mixed precision for speed and memory savings, scheduling the learning rate with warmup and cosine decay, and checkpointing both the best and periodic model states. After that, it broadens to the scaling story behind large language model training, first using gradient accumulation to simulate large batches on limited memory, then comparing data parallelism approaches such as DP and DDP, and finally introducing model parallelism and pipeline scheduling. It shows why naive pipeline schedules waste GPU time, why 1F1B improves utilization and memory use, and how DeepSeek’s DualPipe and DualPipeV ideas further hide MoE communication overhead by overlapping computation and communication in a folded, V-shaped pipeline design.

This chapter is dedicated to part 2 of Stage 3, where we take the core architecture from Stage 2, combine it with advanced training techniques like MTP and FP8, and build a complete foundation model. We will also introduce the final training innovation, DualPipe Parallelism.

The two primary types of parallel computing in deep learning. Data Parallelism accelerates training by splitting the data, and Model Parallelism bypasses memory limits by splitting the model itself. In modern LLM training, these are not either/or choices, but complementary techniques combined to simultaneously overcome the dual constraints of time and hardware.

The workflow of Data Parallelism. A large data batch is sharded into mini-batches, and each GPU processes one mini-batch using its local copy of the model.

The forward pass in a naive pipeline schedule. Each GPU must wait for the previous one to complete its work on a micro-batch. The white area to the left of the bars represents the initial idle time (the "bubble") as the pipeline fills up.

The backward pass in a naive pipeline schedule. The process moves in reverse, starting from the last GPU. The white area to the right of the bars represents the idle time as the pipeline drains.

The combined GPipe schedule, showing the pipeline bubble. The large block of idle time (white space) between the forward and backward passes represents a massive amount of wasted computational resources.

The 1F1B interleaved schedule. Forward (blue) and backward (green) passes alternate during the steady state. Note: any small gaps visible in this rendered timeline are artifacts of the profiling resolution, not the schedule itself. In a true 1F1B, each GPU executes its remaining backward passes back-to-back during cool-down.

The DualPipe schedule with fine-grained overlap. This diagram shows how forward, backward, and overlapped tasks are scheduled to keep the GPUs constantly busy and hide communication latency.

The V-Shape Schedule of DualPipeV. Micro-batches flow down through the first half of the stages and then up through the second half, minimizing the bubble while using only N/2 GPUs for an N-stage pipeline.

• Building a complete training pipeline involves preparing a dataset, defining the model architecture, writing a training script to optimize the model, and a sampling script to generate text.
• Gradient Accumulation is a technique that simulates a large, effective batch size on memory-constrained hardware by processing smaller mini-batches sequentially and accumulating their gradients before performing a single weight update.
• Data Parallelism accelerates training by replicating the entire model on multiple GPUs, allowing each device to process a different shard of the data batch in parallel.
• The combination of Data Parallelism and Gradient Accumulation is the foundational strategy for scaling training, solving both the time and memory bottlenecks for models that can fit on a single GPU.
• Pipeline Parallelism is a model parallelism technique that solves the "model-is-too-big" problem by splitting the model's layers into stages and distributing them across multiple GPUs.
• The naive GPipe schedule is inefficient due to the "pipeline bubble," a period of significant GPU idle time caused by waiting for the pipeline to fill and drain.
• The 1F1B interleaved schedule drastically reduces the pipeline bubble by alternating forward and backward passes, but its efficiency is hampered in MoE models by the All-to-All communication bottleneck.
• DualPipe Parallelism, DeepSeek's innovation, uses a "folded" pipeline layout and fine-grained scheduling to overlap computation with MoE communication, enabling highly efficient training for massive MoE models at the cost of increased parameter memory per GPU.

FAQ