3 Choosing What to Quantize and at What Granularity
This chapter develops a practical way to decide what to quantize and how finely to quantize it. It emphasizes that neural networks contain three very different targets: frozen weights, input-dependent activations, and, in transformers, the KV cache that grows during generation. Because these objects differ in stability, outlier behavior, and reuse patterns, they do not benefit equally from the same precision scheme. The chapter’s core message is to allocate bits where they matter most, rather than applying one uniform quantization rule everywhere.
For weights, the chapter shows why per-channel quantization is usually superior to per-tensor quantization. Weights are static and generally centered near zero, so they are well suited to symmetric quantization and can often tolerate INT8 or even INT4. Per-tensor scaling is simple but wastes range when some channels contain much smaller values than the global maximum, especially in transformer layers. Per-channel scaling largely removes this outlier penalty with minimal metadata overhead, making it the default choice for most linear and convolutional weights, while a few sensitive tensors such as embeddings, classification heads, and biases are often left in higher precision.
Activations and KV cache entries require more nuanced treatment because their statistics change or accumulate over time. Activations are input-dependent, heavy-tailed, and often dominated by outliers, so calibration is necessary and dynamic or per-token schemes can be much more robust than static per-tensor quantization. The KV cache is special: keys show stable channel-wise structure and are best quantized per-channel, while values vary more by token and are better quantized per-token. The chapter closes with a broader view of group and block quantization for low-bit weights, explaining how memory layout, packing, and hardware support influence the best granularity, and it distills everything into a checklist: choose the target, match granularity to its structure, then verify the scheme against model sensitivity and runtime constraints.
ResNet-18 first conv layer weight distribution
Most channels in ResNet-18's conv1 have maximum weights well below the global max of 1.016 (left), using only 20–50% of the integer range under per-tensor quantization (right)—the inefficiency that per-channel quantization eliminates.
Per-channel quantization reduces MSE by up to 56,000× for underutilized channels (left). The largest gains (right) align with the near-zero channels from Figure 3.2 that collapsed to a single integer under per-tensor quantization.
BERT's query projection weights cluster around 0.15 while the global max reaches 0.525, leaving 93% of channels below 50% integer utilization (top). The error distributions (bottom) tell the story: per-tensor produces flat, uniform error across ±0.002; per-channel concentrates errors near zero, reducing MSE by 9×.
The outlier dilemma quantified. Protecting the max (25.8) preserves outliers perfectly but leaves only 9 integer levels for typical values. Clipping to the 99th percentile (4.8) gives typical values 51 levels—6× better resolution—but introduces catastrophic error (21!) for the clipped outliers.
Left panel: Heatmap or bar chart showing "Hottest Channel Ratio" (max/median) per layer, highlighting layer 10's extreme 36.1x ratio Right panel: Bar chart showing "Number of Outlier Channels (>5x median)" per layer, showing the concentration in layers 3-5
Attention dataflow with KV cache highlighted. Read left-to-right: (1) each token's hidden state projects into Q, K, V vectors; (2) QK^T dot products produce raw relevance scores; (3) softmax normalizes scores into attention weights; (4) weights scale V vectors, and the weighted sum becomes the output. The dashed box marks the KV cache—keys and values from all prior tokens, stored to avoid recomputation. For quantization, the critical observation is that K and V sit on different sides of the softmax, leading to different statistical structures.
Row-major layout means input features are consecutive in memory (left, arrow shows memory order). Grouping along this dimension (right, each color = one scale factor) enables coalesced loads: read G weights, load one scale, accumulate. Grouping along output features would scatter access across memory.
INT4 packing stores two values per byte. The first value occupies the high nibble (bits 4–7), the second the low nibble (bits 0–3). Unpacking requires bit shifts and masks before arithmetic—a cost that must be offset by memory bandwidth savings.
Smaller groups reduce MSE (left) but add scale overhead that erodes compression (right). Group-128 sits at the sweet spot: ~3% overhead for 2× lower error than group-256. Below g=64, overhead climbs steeply with diminishing accuracy returns.
Granularity decision tree. INT8 is simple: per-channel is the standard. INT4 requires choosing your priority—group-128 balances accuracy and compression, smaller groups maximize accuracy at higher overhead, larger groups maximize compression at some accuracy cost.
End-to-end quantization decision tree, read top-to-bottom. Phase 1 (top): select which tensors to quantize—weights are always included, activations add compute speedup, KV cache is context-length dependent. Phase 2 (middle): for each target, select granularity using bit-width as the primary fork—INT8 paths favor per-channel/per-token, INT4 paths favor group-128. Phase 3 (bottom): architecture-specific overrides (e.g., first/last layers in FP16). All paths converge on the hardware compatibility gate (right)—if your runtime doesn't support the chosen scheme, fall back to the next coarser granularity.
Summary
- Weights are the safest target—static, zero-centered, tolerant of perturbation. Even INT4 often preserves quality with appropriate granularity.
- Activations unlock compute speedup but require calibration. Static works for CNNs; dynamic per-token is essential for transformers.
- The KV cache dominates long-context LLM memory, exceeding model weights at 128K context. Keys need per-channel quantization; values need per-token.
- Per-channel is the INT8 standard (3–9× better MSE than per-tensor). Group-128 is the INT4 standard (~2× better than per-channel, ~3% overhead).
- Memory layout must align with granularity—group along the contiguous axis for coalesced loads.
- The decision framework: select targets, choose granularity by bit-width, apply architecture-specific adjustments, verify hardware compatibility.
Quantization and Fast Inference ebook for free
