Overview
8 Pushing into Low Bits Safely
The chapter explains when it makes sense to move below 8-bit quantization and frames the decision around the binding deployment constraint: faster inference on modern accelerators, trainable fine-tuning on limited consumer GPUs, or CPU-only deployment with no multiply-accumulate on the hot path. It argues that these pressures lead to three distinct format families—low-bit floats like FP8 and FP4, distribution-matched codebooks like NF4, and ternary weights—and that each is useful only in the setting it was designed to solve.
FP8 is presented as the practical path for high-throughput inference and mixed-precision training on Hopper-class hardware, with E4M3 favored for weights and activations because it concentrates precision near zero, and E5M2 favored for gradients because range matters more there. The chapter emphasizes that scaling is mandatory, especially for activations, and that blockwise scaling substantially improves fidelity over per-tensor scaling. FP4 pushes the idea further, but its tiny value budget makes block scaling even more essential; with proper blockwise scaling, FP4 can recover near-BF16 quality at much lower memory cost, especially on Blackwell-class hardware where FP4 tensor cores are native.
NF4 is shown as the best choice for QLoRA-style fine-tuning on consumer GPUs because it matches the normal-shaped distribution of neural weights and keeps a frozen 4-bit base trainable through low-rank adapters. Ternary quantization, by contrast, is shown to fail badly as post-training conversion: rounding a pretrained model to {−1, 0, +1} destroys coherence. Ternary works only when the model is trained with that constraint from the start or adapted with quantization-aware training, as in BitNet. The chapter closes by consolidating the results into a Pareto view: the right format is the one that best fits the actual constraint, while methods from earlier chapters remain the better choice when sub-8-bit pressures are absent.
A Sub-8-bit format decision framework. Start from the binding deployment constraint and follow the branches. The three main paths correspond to the three pressures: throughput (FP8), fine-tuning memory (NF4), and CPU-only deployment (ternary). The ternary branch has three gates: native training from scratch (BitNet b1.58, best quality), quantization-aware fine-tuning from a BF16 checkpoint (moderate quality, demonstrated by TII and HuggingFace), and neither (fall back to INT4 GPTQ/AWQ with GGUF packaging). Weight methods from Chapter 7 and KV cache methods (TurboQuant, Section 7.5) remain orthogonal—combine them with any format above.
Representable values in [0, 8]. E4M3 and E5M2 cluster their values exponentially—dense near zero, sparse at larger magnitudes. INT8 symmetric spaces all 128 positive values uniformly, so its coverage ends abruptly at the scale-dependent maximum (here ≈2, assuming scale = 0.0157). Beyond that boundary, INT8 cannot represent any value at all.
Near-zero density in [0, 0.5]. E4M3 packs 49 representable values into this interval; E5M2 packs 57 (more exponent range means more subnormal granularity); INT8 symmetric fits only 32. Neural network weight and activation distributions peak sharply near zero, so denser coverage here translates directly to lower quantization error where it matters most.
Activation quantization error (reconstruction MSE, log scale) across transformer layers. Block-32 FP8 reduces MSE by 1.4–25.9× compared to per-tensor FP8. The improvement is largest in early layers (layers 0–1, 21–26×), where outlier activations are most concentrated, and narrows in later layers (layers 24–31, 1.4–1.5×) as activation distributions become more uniform. INT8 per-tensor is consistently the worst, confirming that FP8's non-uniform spacing outperforms uniform spacing even at the same scaling granularity.
Weight reconstruction MSE by format (log scale). INT8 per-channel achieves the lowest MSE (≈3.5×10⁻⁸ average), followed by FP8 E4M3 (≈2.1×10⁻⁷, roughly 6× worse), then FP8 E5M2 (≈8.5×10⁻⁷, roughly 4× worse than E4M3). INT8 wins here because per-channel scaling gives each output channel its own scale factor—fine-grained enough to neutralize FP8's density advantage.
FP8 inference: memory and quality on OPT-6.7B. Weight-only FP8 E4M3 quantization halves model storage from 12.40 GB to 6.21 GB (2.00× compression). Perplexity is unchanged: 16.26 for both BF16 and FP8. On hardware with FP8 tensor cores (H100/Blackwell), the torch._scaled_mm kernel executes the matmul directly in FP8, achieving cosine similarity of 0.9997 versus the BF16 reference.
Representable values in [0, 6]. FP4 E2M1 and INT4 symmetric each have 8 non-negative values (including zero), but FP4 clusters them toward the origin while INT4 spaces them uniformly. FP8 E4M3 (bottom row) packs 78 values into the same interval—roughly 10× the resolution of either 4-bit format.
Near-zero density in [0, 2]. FP4 places 5 values in this range compared to INT4's 3, giving FP4 a 67% density advantage in the interval where most neural network weights concentrate. FP8 E4M3 packs 65 values into the same range—13× denser than FP4. The density gap between 4-bit and 8-bit formats is the fundamental cost of halving the bit width.
FP4 E2M1 weight reconstruction MSE by block size (log scale). Per-tensor scaling (tallest bars, cross-hatched) produces MSE of ≈1×10⁻⁵ across all layers, rising to 2×10⁻⁵ in layers 30–31. Block-16 (shortest bars, solid fill) reduces MSE by 3.3–6.2× depending on the layer. Each halving of the block size (128→64→32→16) yields a consistent incremental improvement. The gap between per-tensor and block-16 is largest in layers 30–31, where weight magnitude variation is most pronounced.
Weight quantization error across formats (log scale). Five configurations compared on OPT-6.7B fc1 layers. From worst to best: INT4 per-tensor (tallest bars, horizontal hatch) at 2.80×10⁻⁵ average MSE, FP4 per-tensor (cross-hatched) at 1.00×10⁻⁵, INT4 group-128 (diagonal hatch) at 4.56×10⁻⁶, FP4 block-16 (solid fill) at 2.65×10⁻⁶, and FP8 E4M3 per-tensor (shortest bars, vertical hatch) at 2.14×10⁻⁷. FP4 block-16 beats INT4 group-128 by 1.7× despite both being 4-bit formats.
FP4 inference: memory and quality on OPT-6.7B. Left: model size in GB. Right: WikiText-2 perplexity. Per-tensor FP4 (cross-hatched, second bar from left) compresses to 3.11 GB but perplexity spikes to 26.09—a 60% degradation. Block-16 FP4 with FP32 scales (solid fill) recovers to 16.49 (+1.4%) at 4.66 GB. Block-32 FP4 (diagonal hatch) achieves 16.54 (+1.7%) at 3.89 GB. The BF16 baseline (dotted fill) is 12.40 GB at 16.26 perplexity.
NF4, INT4 uniform, and FP4 E2M1 quantization levels in [-1, 1]. NF4 (top row) clusters its levels near the center, matching the normal distribution's density. INT4 (middle) spaces levels uniformly. FP4 (bottom) clusters near zero via its exponential spacing. NF4 and FP4 each place 7 levels in [-0.3, 0.3]; INT4 places only 4.
Near-zero density: levels in [-0.3, 0.3]. NF4 and FP4 each provide 7 levels in this interval; INT4 provides 4. The denser coverage directly reduces quantization error where the majority of weight values land.
Weight reconstruction MSE across formats on OPT-6.7B fc1 layers (log scale). NF4 block-64 (shortest solid bars) achieves the lowest 4-bit MSE at 2.64×10⁻⁶ average. INT4 group-64 (diagonal hatch, same block size) is 1.44× worse. INT4 group-128 (GPTQ/AWQ default) is 1.73× worse. FP4 block-16 (backslash hatch) matches NF4 at 2.65×10⁻⁶ but uses 4× finer blocks. FP8 E4M3 per-tensor (vertical hatch, shortest bars) remains 12× better than any 4-bit format.
QLoRA memory breakdown for Llama-2-7B. The NF4 base occupies 3.6 GB; LoRA adapters add 0.6 GB; optimizer states, gradients, and activations account for 8.1 GB during training. The peak of 12.3 GB fits below the T4's 16 GB limit (dashed line) and well below the RTX 4090's 24 GB (dash-dotted line).
QLoRA training loss over 200 steps (Llama-2-7B, NF4, LoRA rank 16, cosine schedule). Loss decreases steadily from 1.63 to 0.68, confirming that the model is learning despite the frozen 4-bit base. The slight plateau around steps 75–110 coincides with the learning rate peak before cosine decay begins.
Representable values in [−1, 1]. Ternary (top row) has 3 levels — only the endpoints and zero. NF4 (16 levels) and FP4 E2M1 (15 levels) pack 7 values each into the [−0.3, 0.3] interval where most neural network weights concentrate. INT4 (16 levels) places 4. Ternary places 1. The visual gap between 3 levels and 15–16 is the cost of halving the bit width twice.
Ternary PTQ perplexity on OPT-6.7B (log scale). The BF16 baseline (leftmost bar) is 16.30. All three ternary PTQ methods produce perplexities above 13,000 — the model is no longer producing coherent text. Naive per-tensor is the worst at 38,661. TWN per-tensor (13,730) and TWN per-row (14,576) are 840–890× worse than baseline. For comparison, NF4 block-64 achieves 16.49 on the same model (Section 8.4) — a 1.2% degradation, not an 84,000% one.
QAT vs PTQ on a toy MLP (64→256→32, 500 steps). The FP32 student (purple) converges to test MSE 0.0052. BitLinear QAT (green) converges to 0.0070 — 1.4× worse, a modest cost for ternary weights. Post-hoc ternarizing the trained FP32 model (dashed red line) yields 0.0199 — 3.8× worse. The model that trained with the constraint absorbs it; the model that never saw it breaks.
BitNet b1.58 2B4T vs OPT-2.7B. Left: model storage. The BF16 master weights (middle bar, 4.49 GB) are what HuggingFace transformers loads for correct inference. The packed format (right bar, 0.45 GB) is what bitnet.cpp deploys. Right: WikiText-2 perplexity is comparable across models despite different tokenizers (19.22 vs 20.27).
Sub-8-bit Pareto frontier on OPT-6.7B (WikiText-2, log-scale perplexity). The frontier runs BF16 (12.40 GB, 16.26) → FP8 E4M3 (6.21 GB, 16.26) → NF4 block-64 (3.77 GB est., 16.49) → FP4 block-32 (3.89 GB, 16.54). FP4 block-16 (4.66 GB, 16.49) is dominated by NF4 on size at equal quality. FP4 per-tensor (3.11 GB, 26.09) is smaller than any other point but functionally broken. Ternary configurations are excluded: OPT-6.7B has no trained ternary checkpoint, and ternary PTQ produces perplexities above 13,000 (Section 8.5.1). NF4 size is computed from the 4.5 effective bits per weight reported in Section 8.4; all other sizes are measured.
Summary
- When throughput is the binding constraint on H100 or later, FP8 is the answer. torch._scaled_mm runs the matmul in FP8 tensor cores at twice the BF16 rate; OPT-6.7B halves in size at zero perplexity loss. Use E4M3 for forward-pass tensors, E5M2 for gradients; block scaling on activations, per-tensor on weights. Without H100 or later, FP8 saves memory but not compute; the hardware gate matters.
- When fine-tuning memory is the binding constraint on a consumer GPU, NF4 with QLoRA is the answer. NF4 is a 16-entry codebook at normal quantiles, not a numeric format, and it achieves 1.44× lower reconstruction MSE than INT4 at matched scaling granularity. The QLoRA configuration for Llama-2-7B peaks at 12.31 GB during training, fitting a 16 GB T4 where full FP16 fine-tuning would need an A100. NF4 has no hardware acceleration on any GPU; dequantization happens in bitsandbytes CUDA kernels before the matmul runs in BF16.
- When the hardware budget is zero GPUs, ternary is the answer, but only with training access. Post-training ternary quantization destroys OPT-6.7B (perplexities above 13,000), because three representable values cannot approximate a continuous weight distribution regardless of scaling. BitLinear solves this by quantizing to {−1, 0, +1} during training with a straight-through estimator; BitNet b1.58 2B4T demonstrates the result: 20.27 perplexity at 0.45 GB on bitnet.cpp, CPU-only addition and subtraction. Without training access, INT4 GPTQ/AWQ with GGUF packaging remains the practical CPU path.
- At 4 bits, scaling granularity is the dominant lever. FP4 per-tensor degrades OPT-6.7B perplexity by 60%; FP4 block-16 recovers to within 1.4% of BF16. INT4 per-tensor is worse than FP4 block-16 by more than 10×. Groupwise or blockwise scaling at 4 bits is a correctness requirement, not an optimization.
- Non-uniform spacing helps less than scaling granularity does. FP4 block-16 only matches NF4 block-64 because it uses 4× finer blocks. At matched granularity, NF4 beats INT4 by 1.44× and edges FP4 by 1.09×. The codebook matters; the scale factors matter more.
- Hardware support gates format choice. FP8 requires H100 or later; FP4 requires Blackwell; NF4 runs anywhere but accelerates nothing; ternary runs on CPUs through bitnet.cpp with no mainstream GPU kernel. Matching format to silicon is not optional.
- Every inference number in the chapter comes from OPT-6.7B on WikiText-2 (64 sequences × 512 tokens); QLoRA numbers come from Llama-2-7B on Alpaca. The companion scripts reproduce each figure and table, gating hardware-specific paths on torch.cuda.get_device_capability() with graceful fallback.
Quantization and Fast Inference ebook for free
