4 Applying Post-Training Quantization and Calibration
Post-training quantization is presented as a fast, practical way to compress models without retraining, but its success depends on how sensitive the model is, how aggressively it is quantized, how much redundancy it has, and how much accuracy loss the application can tolerate. The chapter emphasizes that INT8 quantization is usually the easiest path, while INT4 and below often require more careful grouping or even quantization-aware training. Architecture matters too: convolutional models with robust representations tend to withstand quantization better than attention-heavy transformer models, which are more prone to outlier-driven error accumulation.
A major theme is that calibration quality is as important as the quantization method itself. Calibration data must reflect production conditions, not merely resemble training data in a superficial way, because mismatched inputs can produce poor scale estimates and silent accuracy loss in deployment. The text shows that representative calibration means covering the activation space across difficult cases, shifts in usage, and domain-specific variation, and that only a few hundred diverse samples are often enough if they expose the true extremes. The chapter also argues for validating calibration against real or shadow production traffic so range estimates can be corrected before quantization is shipped.
For range estimation, the chapter compares absolute maximum, percentile clipping, MSE-optimal search, and KL-divergence methods, showing that no single approach is best for every distribution. Absolute maximum is simple and safe against clipping but can waste precision on outliers, percentile methods often strike a good balance, MSE-optimal calibration tends to work best for heavy-tailed transformer activations, and KL-based entropy calibration can fail badly when tails are extreme. The final validation workflow confirms that INT8 compression reliably gives about 4× smaller models, but accuracy depends strongly on calibration choice and model type; in the examples, MSE-optimal preserved accuracy best, while runtime speedups still require a true INT8 execution backend rather than just storing weights in low precision.
Left: Signal-to-Noise ratio by bit-width with PTQ viability zones. Right: Signal-to-noise ratio as intuitive ratios
PTQ decision framework flowchart. Starting from the target bit-width at the top, the reader follows branching decisions through model architecture checks, calibration method selection, and accuracy validation gates. Each terminal node recommends either shipping the PTQ result or escalating to QAT.
Workflow for preparing a calibration set. The process flows from top to bottom: source production data from logs or traffic samples, apply stratified sampling to cover user segments and edge cases, validate distribution coverage against known activation patterns, and iterate if coverage gaps are found. Each stage includes a quality gate that prevents poorly representative data from reaching calibration.
Left: Sample images under four conditions (original, low-light, high-contrast, noisy) showing the visual transformations applied to STL-10 images. Right: Layer-wise activation maximums through ResNet-18, where each line corresponds to one condition. Notice how low-light images produce consistently lower activation ranges while high-contrast images push activation maximums higher in deeper layers—calibrating on only one condition would systematically mis-estimate the ranges encountered in production.
Comparison of range estimation methods on BERT-base layer 6 activations
Decision framework for range estimation method selection. Start by computing the outlier ratio (max / 99th percentile) on your calibration data. If the ratio is below 2×, activations are well-behaved and absolute maximum (absmax) is sufficient. Between 2–10×, percentile clipping at 99.99% provides a safe default. Above 10×, use MSE-optimal calibration to automatically find the best clipping threshold. The framework routes you to the simplest method that handles your distribution shape.
MSE calibration preserves accuracy best (top-left); MSE-optimal achieves lowest quantization error across architectures (bottom-right)
Summary
- PTQ works best for INT8 quantization of CNNs and BERT-scale transformers with smooth weight distributions—it delivers 4× size reduction with minimal accuracy loss and zero retraining.
- Calibration data determines quantization ranges, so your calibration set must match production data distribution. A few hundred representative samples usually suffice, covering edge cases and deployment scenarios.
- MSE-optimal range estimation delivered the best accuracy in our experiments (ResNet: -0.20%, BERT: 0.00% loss), though absmax works for clean distributions and percentile helps with sparse outliers.
- Size reduction is guaranteed (4× for INT8 is arithmetic), but accuracy preservation depends on calibration quality—always validate against FP32 baseline before deployment.
- Latency improvement requires runtime support (PyTorch quantization backends, TensorRT, or ONNX Runtime)—storing INT8 weights alone won't speed up inference without proper execution backends.
- When PTQ fails to meet accuracy targets, the path forward is Quantization-Aware Training (Chapter 5) for strict requirements or specialized LLM techniques (Chapter 7) for sub-8-bit precision.
Quantization and Fast Inference ebook for free
