9 Advanced Quantization Techniques

Advanced quantization techniques become important when a domain-specific language model is no longer small enough for the target hardware, even after applying standard quantization methods. The chapter explains that specialized models are often manageable in constrained environments, but larger training datasets, complex unstructured inputs, or the choice of a larger base model can push memory and inference requirements beyond practical limits. It presents advanced strategies that help balance model quality, speed, cost, and deployability, especially when conventional approaches are not sufficient.

FlexGen is introduced as a way to run very large language models with limited GPU memory by distributing weights, activations, and attention cache across GPU, CPU memory, and disk. Its optimizer chooses efficient storage and access patterns, and it can compress weights and KV cache to 4 bits to improve throughput with minimal accuracy loss. The chapter emphasizes that FlexGen is useful for throughput-oriented workloads and large batch inference, but it can add latency because of offloading and extra I/O operations. SmoothQuant addresses a different problem: activation outliers in larger models that can severely damage accuracy after naive 8-bit quantization. By smoothing activations and shifting quantization difficulty to weights, it enables effective W8A8 quantization without retraining, preserving accuracy while reducing memory usage and improving inference efficiency.

BitNet is presented as a more architectural approach to low-bit modeling, using 1-bit weights through BitLinear layers while keeping higher precision where needed for stability and output quality. Unlike post-training quantization methods, BitNet is designed to train low-bit Transformers from scratch and aims to reduce memory consumption, energy use, and inference costs. The chapter notes that BitNet becomes more competitive as model size grows, can show strong optimization stability, and may outperform several post-training quantization methods while remaining slightly behind full FP16 baselines. Overall, the chapter concludes that while traditional quantization is often enough for many local small language model deployments, FlexGen, SmoothQuant, and BitNet provide valuable options for larger, more demanding, or resource-constrained production scenarios.

The hardware needed to load into memory a 175 billion language model

Loading a 175 B language model using FlexGen across GPU, CPU and disk.

High-level explanation of the idea behind SmoothQuant.

The BitLinear component (left) and the BitNet architecture (right).

Comparing the scaling curves for BitNet and FP16 Transformers.

The results of the stability tests for BitNet and FP16 Transformers.

Comparison of the average accuracy on zero-shot (a) and few-shot (b) on downstream tasks for vanilla FP16 Transformers, BitNet and diverse PTQ techniques.

Zero-shot results for BitNet, the Transformer baseline and after quantization with different techniques.

• FlexGen offloads model weights, activations, and attention cache across GPU, CPU, and disk to enable large model inference on limited hardware.
• FlexGen uses linear programming optimization to find efficient patterns for storing and accessing tensors across different storage types.
• FlexGen compresses weights and attention cache to 4 bits with minimal accuracy loss to increase throughput.
• Policy class in FlexGen configures offloading percentages and compression settings for weights, cache, and activations.
• FlexGen prioritizes throughput over latency and works best for batch processing scenarios with millions of tokens.
• SmoothQuant addresses activation outliers in large language models by migrating difficult quantization from activations to weights.
• SmoothQuant applies mathematical transformations to smooth activation magnitudes across channels before quantization.
• Models above 6-7 billion parameters develop systematic activation outliers that degrade quantization performance.
• SmoothQuant preserves accuracy during 8-bit quantization of both weights and activations for large models.
• W8A8Linear replaces standard linear layers to enable 8-bit weight and activation quantization in SmoothQuant.
• BitNet introduces BitLinear components that replace traditional linear layers with 1-bit weight representations.
• BitNet maintains high precision for optimizer states and gradients while using 1-bit weights in attention layers.
• BitLinear components reduce computational costs while preserving precision for input/output embeddings.
• BitNet shows better training stability and convergence compared to traditional FP16 Transformers at high learning rates.
• BitNet achieves competitive performance with significantly lower memory usage and energy consumption than 8-bit quantization.
• Python BitNet implementations provide PyTorch-compatible replacements for linear layers in existing models.

FAQ