Overview

2 Building Quantization from First Principles

This chapter explains quantization as the bridge between floating-point training and integer-based inference hardware. It begins with fixed-point numbers, showing how hardware stores integers that are interpreted through an imaginary binary point, and uses simple examples to make two’s complement, wraparound behavior, and saturation intuitive. From there, it motivates why deep learning needs more flexibility than rigid fixed-point rules can provide, especially when tensors vary widely in scale and shape.

The core mathematical development is the affine quantization map, which introduces a scale and zero-point so each tensor can use a custom integer grid. The scale sets the step size or resolution, while the zero-point shifts the grid so real zero is exactly representable, which is crucial for padding and sparsity. The chapter also derives how these parameters are chosen from data ranges, shows how quantization and dequantization work in practice, and compares per-tensor, per-channel, and group-wise granularity choices as different ways to balance accuracy against implementation cost.

The chapter then analyzes the two main sources of error: granular error from rounding values that fall between grid points, and overload error from clipping values outside the representable range. These errors trade off against one another, so calibration is really about choosing where distortion should occur. Finally, it compares symmetric and asymmetric schemes, arguing for a hybrid approach in which weights are usually symmetric for efficiency while activations are often asymmetric for better use of the available codes. End-to-end PyTorch examples confirm that this combination can closely match full-precision results, while also showing how zero-point correction becomes essential when activations are shifted away from zero.

The Q3.4 fixed-point format divides 8 bits into a sign bit, 3 integer bits, and 4 fractional bits. Here, the bit pattern 0 001 1000 encodes +1.5 — the integer bits 001 contribute 1, and the leading fractional bit 1000 contributes 2-¹ = 0.5.
The Q3.4 fixed-point format divides 8 bits into a sign bit, 3 integer bits, and 4 fractional bits. Here, the bit pattern 0 001 1000 encodes +1.5 — the integer bits 001 contribute 1, and the leading fractional bit 1000 contributes 2-¹ = 0.5.
The 8-bit signed integer ring. The 256 possible bit patterns wrap around a circle, with 0 at the top and −1 (1111 1111) one step below it. Moving clockwise increments the integer; moving counterclockwise decrements it.
The 8-bit signed integer ring. The 256 possible bit patterns wrap around a circle, with 0 at the top and −1 (1111 1111) one step below it. Moving clockwise increments the integer; moving counterclockwise decrements it.
The range-versus-precision tradeoff in Q-format. For a fixed 8-bit container, every bit allocated to the integer part doubles the representable range but halves the precision (step size). Moving the binary point left (more fractional bits) gives finer resolution at the cost of a narrower range; moving it right does the reverse. There is no free lunch: the total information capacity is fixed by the bit-width.
The range-versus-precision tradeoff in Q-format. For a fixed 8-bit container, every bit allocated to the integer part doubles the representable range but halves the precision (step size). Moving the binary point left (more fractional bits) gives finer resolution at the cost of a narrower range; moving it right does the reverse. There is no free lunch: the total information capacity is fixed by the bit-width.
The ruler problem. Fixed-point quantization (left) forces a single rigid ruler on all measurements: a rice grain vanishes between coarse tick marks, while a watermelon extends beyond the ruler's edge and gets clipped. Affine quantization (right) manufactures a custom ruler for each object, with tick spacing (scale) and starting position (zero-point) tailored to the data's actual range.
The ruler problem. Fixed-point quantization (left) forces a single rigid ruler on all measurements: a rice grain vanishes between coarse tick marks, while a watermelon extends beyond the ruler's edge and gets clipped. Affine quantization (right) manufactures a custom ruler for each object, with tick spacing (scale) and starting position (zero-point) tailored to the data's actual range.
The granular-versus-overload error tradeoff. Widening the quantization grid (increasing the clipping range) reduces overload error—fewer values are clipped—but coarsens the step size, increasing granular error for values inside the range. Tightening the grid does the reverse. The optimal clipping boundary minimizes total distortion and depends on the shape of the data distribution: heavy-tailed distributions push the boundary outward; tightly concentrated distributions pull it inward.
The granular-versus-overload error tradeoff. Widening the quantization grid (increasing the clipping range) reduces overload error—fewer values are clipped—but coarsens the step size, increasing granular error for values inside the range. Tightening the grid does the reverse. The optimal clipping boundary minimizes total distortion and depends on the shape of the data distribution: heavy-tailed distributions push the boundary outward; tightly concentrated distributions pull it inward.
Symmetric versus asymmetric quantization. Asymmetric quantization (left) shifts the grid to cover only the region the data actually occupies, reclaiming bits that would otherwise encode empty space. The zero-point Z records the shift, and must be accounted for during inference arithmetic. Symmetric quantization (right) pins the grid at zero and mirrors the range, making it ideal for centered distributions like weight tensors but wasteful for one-sided data.
Symmetric versus asymmetric quantization. Asymmetric quantization (left) shifts the grid to cover only the region the data actually occupies, reclaiming bits that would otherwise encode empty space. The zero-point Z records the shift, and must be accounted for during inference arithmetic. Symmetric quantization (right) pins the grid at zero and mirrors the range, making it ideal for centered distributions like weight tensors but wasteful for one-sided data.
Weight quantization (symmetric)
Weight quantization (symmetric)
Activation quantization (affine/asymmetric)
Activation quantization (affine/asymmetric)
Symmetric vs asymmetric on ReLU activations
Symmetric vs asymmetric on ReLU activations
FP32 baseline vs INT8 output
FP32 baseline vs INT8 output

Summary

  • Fixed-point arithmetic stores numbers on a rigid discrete grid determined at design time. The Q-format notation makes explicit the fundamental bargain: allocate bits to range or to precision, but never both. Two's complement representation enables negative numbers through overflow, while saturation at the overflow cliff constrains every fixed-point system.
  • The affine transformation r = S(q - Z) bridges floating-point tensors and integer storage. Scale (S) is the ratio of real range to integer range; zero-point (Z) aligns real zero with an integer coordinate. The zero-point nudge forces the calibrated range to include 0.0, preserving sparsity after ReLU and enabling correct zero-padding.
  • Granular error arises when values land between grid points and must be rounded, bounded by S/2. Overload error arises when values fall outside the grid and are clipped, introducing unbounded distortion. Widening the grid reduces clipping but increases rounding; tightening it does the reverse.
  • Symmetric quantization centers the grid on zero, wasting capacity on one-sided distributions but eliminating cross-term overhead. Asymmetric quantization shifts the grid to follow data, reclaiming bits at the cost of runtime correction. The Hybrid Consensus—symmetric weights, asymmetric activations—matches each scheme to its statistical reality.
  • Visualization reveals wasted bits immediately: if half the integer codes cover empty space, the wrong scheme was chosen. The quantized arithmetic, when implemented correctly with zero-point correction, matches FP32 output to within the noise floor of 8-bit representation.
  • Calibration is not a one-time event. Scale and zero-point derivations in this chapter assumed known min/max—true for weights but not activations, whose ranges shift with every batch. Strategies like percentile clipping and entropy minimization replace raw min/max with robust boundaries; when input distributions drift, recalibration may be needed.

FAQ

What is fixed-point representation in the context of quantization?Fixed-point representation stores numbers as integers and interprets them as real values by dividing by a constant scale factor. In Q-format, the hardware holds an integer, and the decimal point is implicit through that scale.
Why does Q-format use powers of two for the scale factor?In binary hardware, the scale is typically a power of two because it can be implemented efficiently with shifts. For example, Q3.4 uses a scale of 2^-4 = 1/16, so moving between integer and real values is hardware-friendly.
Why does the bit pattern 1111 1111 represent -1 in signed 8-bit two's complement?Two's complement wraps values around a fixed 8-bit ring. In signed 8-bit form, 1111 1111 sums to -128 + 64 + 32 + 16 + 8 + 4 + 2 + 1 = -1, which is why it sits just below zero.
What is the main idea behind affine quantization?Affine quantization maps real values to integers using both a scale and a zero-point. Unlike fixed-point, it can shift the integer grid left or right, which lets it fit asymmetric data distributions more accurately.
How are the scale S and zero-point Z computed?The scale is computed from the real range divided by the integer range: S = (r_max - r_min) / (q_max - q_min). The zero-point is then chosen so that real zero maps to an integer: Z = round(q_min - r_min / S), with clamping to the integer limits.
Why must real zero map exactly to an integer value?Exact zero is essential for zero-padding and sparsity. If zero were not representable exactly, padding would introduce artificial nonzero borders and ReLU zeros would become small noisy values, hurting accuracy and efficiency.
What is the difference between granular error and overload error?Granular error happens when values fall between grid points and are rounded, creating bounded error of at most S/2. Overload error happens when values fall outside the representable range and get clipped, creating potentially much larger, unbounded distortion.
Why does symmetric quantization work well for weights?Weights are usually centered around zero and have a bell-shaped distribution, so a symmetric grid wastes little space. Symmetric quantization also sets the weight zero-point to zero, which removes expensive cross-terms from inference arithmetic.
Why is asymmetric quantization often better for activations?Activations, especially after ReLU, are usually non-negative and skewed. An asymmetric grid shifts to match that range, uses the full integer space more efficiently, and avoids wasting half the codes on negative values that never occur.
What is the Hybrid Consensus in INT8 inference?The Hybrid Consensus uses symmetric quantization for weights and asymmetric quantization for activations. This combines the efficiency of symmetric weights with the better range coverage of asymmetric activations, giving a practical balance of speed and accuracy.

pro $24.99 per month

  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose one free eBook per month to keep
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime

lite $19.99 per month

  • access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more


Close
choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Quantization and Fast Inference Quantization and Fast Inference ebook for free
Close
choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Quantization and Fast Inference Quantization and Fast Inference ebook for free