6 Quantizing for Your Production Environment
This chapter explains how lowering numerical precision can make language models more practical for production deployment, especially when memory, latency, or hardware constraints matter. It begins by comparing common precision formats used in training and inference, including 32-bit floating point, 16-bit floating point, mixed precision, and bfloat16. The main idea is that deep learning models often tolerate reduced precision well, allowing significant memory savings and speed improvements, but the lower the precision becomes, the more carefully accuracy and numerical stability must be managed.
The chapter then introduces quantization as a stronger optimization technique that converts model weights and sometimes activations from floating-point values into low-bit integer formats such as 8-bit or 4-bit. It distinguishes post-training quantization from quantization-aware training, with emphasis on post-training approaches because they are practical for adapting pretrained models. Several 8-bit methods are discussed, including zero-point and absmax quantization, along with their trade-offs between compression and quantization error. Practical examples show how to quantize models with PyTorch and Hugging Face Transformers, how to evaluate generated outputs with perplexity, and how LLM.int8() uses mixed precision to preserve outlier features while reducing memory usage.
The chapter also covers ONNX-based 8-bit quantization and 4-bit quantization workflows for production-oriented inference. It explains low-, mid-, and high-level ONNX conversion paths, then shows how ONNX Runtime and Hugging Face Optimum can dynamically quantize models, reducing model size substantially while preserving accuracy and improving latency. For more aggressive compression, it introduces GPTQ for 4-bit GPU-oriented quantization and ggml-style formats for CPU-friendly inference across common devices. Overall, the chapter presents quantization as a key deployment strategy for making domain-specific language models smaller, faster, and easier to run in constrained production environments while maintaining acceptable performance.
Representation of 32-bit floating point numbers.
Representation of 16-bit floating point numbers.
Representation of bfloat16 numbers.
Comparison between the original GPT-2 model weights and the absmax quantized ones.
Comparison between the original GPT-2 model weights and the LLM.int8() quantized ones
Summary
- Floating point precision determines memory usage and computational speed for language model operations.
- 32-bit floating point uses 1 bit for sign, 8 bits for exponent, and 23 bits for mantissa.
- 16-bit floating point reduces memory requirements by half but may cause performance degradation.
- Mixed precision training switches between 32-bit and 16-bit operations during model training.
- Bfloat16 format provides extended dynamic range compared to float16 at the cost of decreased precision.
- Quantization converts model weights from floating point to low-bit integer representations.
- Post-training quantization applies to trained models while quantization-aware training happens during training.
- Zero-point quantization scales inputs by the total range divided by the difference between maximum and minimum values.
- Absmax quantization divides input values by the absolute maximum value and multiplies by a scaling factor.
- LLM.int8() uses mixed-precision quantization where outliers are processed in fp16 and other values in int8.
- Vector-wise quantization considers variability of values in rows and columns within tensors.
- ONNX runtime provides three levels of quantization implementation from low-level PyTorch to high-level Optimum.
- Dynamic quantization in ONNX reduces model size significantly with minimal performance loss.
- GPTQ enables 4-bit quantization for GPT-based models with negligible accuracy degradation.
- Auto-GPTQ integrates into Hugging Face libraries for streamlined 4-bit quantization workflows.
- GGML library provides CPU-optimized quantization in multiple precisions including 8-bit, 5-bit, and 4-bit.
- GGUF format allows distribution of quantized models as single files with embedded metadata.
- Quantization reduces model memory footprint by 2x to 4x while maintaining comparable performance.
- Perplexity metrics evaluate quantized model performance by measuring prediction uncertainty.
FAQ
What precision formats are commonly used for training and inference in LLMs?
LLMs commonly use several numeric precision formats. Traditional scientific computing often used 64-bit floating point, but deep learning training typically uses 32-bit floating point because it is faster and uses less GPU memory. Mixed-precision training may use 16-bit floating point for parts of computation while keeping higher precision where needed for stability. Another important format is bfloat16, which uses 1 sign bit, 8 exponent bits, and 7 mantissa bits, giving it a wider dynamic range than standard float16 but less precision.
Why is 32-bit floating point preferred over 64-bit floating point for deep learning on GPUs?
32-bit floating point is preferred because it reduces memory usage and increases computation speed. It requires half the memory of 64-bit floating point, allowing larger models or larger batch sizes to fit in GPU memory. GPUs are also better optimized for 32-bit operations than for 64-bit operations, making training faster without usually causing significant loss in model quality.
What is mixed-precision training, and why is it useful?
Mixed-precision training combines lower-precision and higher-precision arithmetic during training. Typically, some computations use 16-bit floating point for speed and memory savings, while gradients or weight updates may be converted back to 32-bit precision for numerical stability. This approach can reduce memory usage and speed up training while preserving model performance. In some cases, the small amount of noise introduced by lower precision can even improve generalization.
What is quantization in the context of LLM deployment?
Quantization is the process of converting an LLM’s weights, and sometimes activations, from floating-point formats into lower-bit integer representations such as 8-bit or 4-bit integers. The goal is to reduce model size, lower memory usage, and potentially improve inference speed. Quantization introduces some numerical error, so the key challenge is balancing efficiency gains against possible accuracy degradation.
What is the difference between post-training quantization and quantization-aware training?
Post-training quantization, or PTQ, is applied after a model has already been trained. It is especially useful when starting from a pre-trained LLM and adapting it through fine-tuning, PEFT, or RAG rather than training from scratch. Quantization-aware training, or QAT, happens during training and teaches the model to account for quantization effects. The chapter focuses mostly on PTQ because it is practical for deploying pre-trained domain-specific LLMs.
How do zero-point and absmax quantization differ?
Zero-point quantization assumes the input values may be asymmetrically distributed. It computes a scale based on the minimum and maximum tensor values, then shifts the scaled values into the target integer range using a zero-point. Absmax quantization uses the maximum absolute value of the tensor and maps values symmetrically into a range such as [-127, 127]. Both techniques are lossy, meaning dequantized values are close to, but not exactly the same as, the original floating-point values.
What are the main benefits and drawbacks of LLM quantization?
The main benefits of LLM quantization are reduced model size, improved memory efficiency, faster inference on supported hardware, and better suitability for deployment on constrained devices. The main drawbacks are possible accuracy loss and implementation complexity. However, modern techniques such as LLM.int8(), GPTQ, ONNX Runtime quantization, and ggml/GGUF can significantly reduce these drawbacks.
What is LLM.int8(), and why is it useful?
LLM.int8() is an 8-bit quantization technique designed to reduce memory usage while preserving near full-precision performance. It uses mixed-precision quantization: most values are processed in int8, while outlier features are processed in fp16 to avoid major accuracy loss. It performs matrix multiplication by separating outliers, computing outliers in fp16, computing non-outliers in int8, then combining the results. It is integrated with Hugging Face Transformers through the Bitsandbytes and Accelerate libraries and can be enabled with load_in_8bit=True.
How can ONNX be used for 8-bit quantization of Transformer models?
ONNX can be used by first converting a Transformer model to an ONNX graph and then applying ONNX Runtime quantization. The chapter describes three conversion levels: low-level conversion with torch.onnx, mid-level conversion with transformers.onnx, and high-level conversion with Hugging Face Optimum. Once converted, dynamic quantization can be applied with ONNX Runtime APIs such as quantize_dynamic and QuantType.QInt8, or through Optimum’s ORTQuantizer.
What results did the chapter show for ONNX 8-bit quantization?
For a BERT model converted to ONNX, the full-precision ONNX file was about 417.91 MB, while the quantized version was about 105.22 MB. In the Optimum example using distilbert-base-uncased-finetuned-banking77, the original model was 255.76 MB and the quantized model was 64.33 MB. The quantized model achieved 92.34% accuracy compared with 92.5% for the original model, retaining about 99.82% of the original accuracy, while inference latency improved by about 1.86x.
What are GPTQ, ggml, and GGUF in 4-bit quantization?
GPTQ is a post-training quantization method that can compress GPT-style models to 3-bit or 4-bit weights with little accuracy degradation. The chapter demonstrates Auto-GPTQ on GPT-2 XL, reducing the model from about 6.43 GB to about 1.02 GB. ggml is a C-based library optimized mainly for CPU inference and supports several quantization levels, including 4-bit. GGUF is a newer format associated with llama.cpp and is becoming a standard for C/C++ quantized models because it packages model data and metadata more conveniently than the older GGML format.
Domain-Specific Small Language Models ebook for free
