5 Multi-token prediction and FP8 quantization

This chapter introduces two techniques that make DeepSeek-style training both stronger and more efficient: multi-token prediction and FP8 quantization. After establishing the model’s major architectural ideas, it shifts focus to how training can be improved by asking the model to predict several future tokens instead of only the next one, and by using low-precision arithmetic in a carefully controlled way. Together, these ideas help the model learn richer sequence structure while reducing the memory and compute cost of training large language models.

Multi-token prediction changes the training objective from a one-step next-token task to a multi-step forecasting task. This creates denser supervision, improves data efficiency, and encourages better long-range planning because the model must account for multiple upcoming tokens from each position. The chapter explains DeepSeek’s causal version of this idea, in which predictions are made through a chain of dependent modules rather than independent heads, so each step refines the hidden state before passing it forward. It also shows how this design can support faster inference through speculative decoding, where multiple candidate tokens are proposed at once and then verified efficiently.

The second half of the chapter explains FP8 quantization as a practical trade-off between precision and performance. It walks through the basics of floating-point formats, why quantization saves memory, and how scaling maps values into lower precision without destroying too much information. DeepSeek’s framework is presented as a set of five supporting ideas: mixed precision computation, fine-grained grouping to reduce the impact of outliers, higher-precision accumulation when needed, a preference for mantissa over exponent bits in FP8 format choice, and online quantization using current tensor statistics. The result is a robust training pipeline that uses FP8 where speed matters most while keeping sensitive parts of the system in higher precision for stability.

This chapter focuses on the highlighted components, Multi-Token Prediction (MTP) and FP8, which are the major innovations in the advanced training pipeline.

The standard single-token prediction process. For a given sequence of input tokens, the model predicts only the single immediate next token for each position.

The Multi-Token Prediction (MTP) process. For a given sequence of input tokens, the model is trained to predict multiple future tokens simultaneously from each position. For example, from the input token "is," it might predict the sequence "transforming," "the," "world."

Performance improvement of MTP over single-token prediction on coding benchmarks. Positive bars indicate MTP is better. (Source: Gloeckle et al., 2024)

The effect of increasing the number of predicted future tokens (n) on benchmark performance. (Source: Gloeckle et al., 2024)

MTP implicitly assigns higher weights to consequential "choice point" tokens. (Source: Gloeckle et al., 2024)

The initial step of the Multi-Token Prediction process. Input tokens are passed through the main "Shared Transformer Trunk," which consists of multiple Transformer blocks. The output is the initial matrix of hidden states (denoted as h⁰ or z), which serves as the starting point for the MTP modules.

The sequential architecture of the DeepSeek MTP modules. The hidden state from one module is passed as an input to the next, forming a causal chain.

The internal operations of a single MTP head.

The Transformer block within a single MTP module. This block takes the merged and projected vector (combining the previous hidden state and the next token's embedding) as input. It then performs a full Transformer computation to produce a new, refined hidden state for the next step in the causal chain.

Equation 22 from the DeepSeek V3 paper.

The final prediction steps within an MTP module. The new hidden state generated by the MTP Transformer block is passed to a "Shared Un-Embedding Matrix." This projects the hidden state into the vocabulary space to produce a logits vector, from which the final token for that prediction step is selected.

Equation 23 from the DeepSeek V3 paper.

The total loss for a single input token is the sum of the individual cross-entropy losses for each of the predicted future tokens.

A visual comparison of a number represented in 32-bit floating-point (FP32) versus 16-bit floating-point (FP16). The diagram illustrates the reduction in the number of bits allocated for the exponent and the mantissa, which results in lower precision and a smaller memory footprint.

The quantized image uses far fewer colors (less information/precision) but still effectively represents the original image.

The memory savings from quantization for a 70-billion-parameter model. The calculations demonstrate how reducing the numerical precision from 64-bits to 32-bits, and further to 16-bits, dramatically decreases the total memory required to store the model's weights.

A comparison of FP32 and FP16.

A comparison of FP32 and BFloat16.

A comparison of FP32 and INT8.

The scaling process for quantization. The original range of the FP32 tensor is mapped to the target range of the INT8 format.

The mixed-precision framework with FP8 data format in the Linear operator. The diagram illustrates the flow of data through a linear layer, showing how inputs and weights are converted to low-precision FP8 for fast computation, while critical components like master weights and gradients are maintained in high-precision FP32 for stability.

The data flow and precision formats for the forward pass.

The data flow for the Dgrad calculation.

The data flow for the Wgrad calculation.

The weight update step is performed entirely in high-precision FP32.

Fine-Grained Quantization for an activation vector. The vector is partitioned into smaller groups. Each group is scaled independently based on the maximum value within that group, preserving precision for values in groups that do not contain large outliers.

Fine-Grained Quantization for a weight matrix. The matrix is divided into smaller blocks (e.g., W₁₁, W₁₂, etc.), and each block is quantized independently with its own unique scaling factor.

The complete Fine-Grained Quantization workflow.

The core strategy for increasing accumulation precision: intermediate results are periodically moved from the low-precision environment of the Tensor Cores to the high-precision environment of the CUDA Cores.

The Increasing Accumulation Precision mechanism in detail. Low-precision accumulation happens in bursts on the Tensor Core, and results are periodically promoted to a high-precision FP32 register on the CUDA Core.

• Multi-Token Prediction (MTP) enhances data efficiency by training models to predict a horizon of future tokens simultaneously, providing richer gradient signals for long-range coherence.
• The MTP training objective implicitly assigns higher weight to "choice points" or critical pattern-shifting tokens in a sequence, thereby developing better planning capabilities in the model.
• DeepSeek's causal MTP implementation sequentially refines a hidden state, where the prediction for one token informs the next, creating a powerful forecasting mechanism.
• Quantization is essential for large-scale training, reducing memory usage and computational cost by converting high-precision parameters to low-precision formats like FP8.
• DeepSeek’s FP8 framework uses a mixed-precision strategy, performing core computations in low precision while storing sensitive components, such as master weights and optimizer states, in high precision to ensure numerical stability.
• Fine-Grained Quantization: To mitigate precision loss from outliers, DeepSeek calculates separate scaling factors for small blocks of activations and weights, preserving fidelity at a local level.
• Increasing Accumulation Precision: DeepSeek promotes intermediate results from low-precision Tensor Cores to high-precision CUDA Cores, where accumulation occurs in full FP32 precision to prevent underflow errors.
• Mantissa over Exponents: DeepSeek uniformly uses the higher-precision E4M3 format for both forward and backward passes, enabled by fine-grained quantization, which effectively manages dynamic range.
• Online Quantization: DeepSeek calculates scaling factors in real-time based on the current batch's data, ensuring accurate scaling and avoiding the instability associated with historical data.

FAQ