6 Porting Workflows Across Frameworks

This chapter explains how quantization work changes once models have to move across frameworks. In a single framework, it is usually enough to pick a granularity, calibrate, and recover accuracy; in production, though, the same model may be trained in PyTorch, served through ONNX Runtime, and deployed on mobile through TensorFlow Lite. The central warning is that identical-looking INT8 workflows can produce different results because frameworks make different choices about quantization semantics, fusion rules, and calibration behavior, so parity must be verified deliberately rather than assumed.

The chapter first grounds that idea in practical end-to-end paths. In PyTorch, TorchAO’s in-place weight-only quantization works well for transformer-heavy models where linear layers dominate, giving strong compression with little or no accuracy loss, but it does not speed up eager-mode inference because computation still runs in FP32. In TensorFlow Lite, the converter provides dynamic-range, full-integer, and FP16 modes; dynamic-range works well for transformers, full-integer is best for CNNs with stable activation ranges, and FP16 offers a low-risk compression option. For ONNX, exporting from PyTorch and then quantizing with ONNX Runtime shows how static quantization can capture CNNs effectively, while dynamic quantization is the right choice for transformers, especially when activation distributions are too variable for frozen calibration scales.

The final emphasis is on validation and diagnosis. The chapter shows that export fidelity can be nearly perfect in FP32, while quantization introduces different degrees of drift depending on model shape, output-space size, and operator coverage. Comparing logits, cosine similarity, and top-1 agreement helps distinguish harmless rounding noise from real behavioral changes, and operator-coverage checks reveal why compression does not always translate to speedup. The overall takeaway is that successful porting means understanding where frameworks disagree, choosing the right quantization path for the architecture, and confirming that the exported artifacts remain numerically equivalent enough for production use.

Three quantization paths from a trained PyTorch model to deployment-ready artifacts. Arrows represent conversion steps: downward arrows feed the FP32 model into each framework's quantization pipeline, horizontal arrows carry the model through quantization and optimization stages, and converging arrows at the bottom feed all three artifacts into a shared validation step. Each path makes independent decisions about quantization semantics, operator fusion, and calibration.

TorchAO INT8 weight-only compression across three architectures. Compression scales with the fraction of parameters in nn.Linear: 1.03× for the Conv2d-dominated ResNet-18, 2.40× for BERT-base, and 3.38× for TinyLlama-1.1B.

ORT quantization results for ResNet-18 on ImageNette. Dynamic quantization (MatMul only) gives 1.03× - identical to TorchAO's Linear-only result. Static quantization (all operators, with calibration) gives 3.95× with 0.10 pp accuracy loss. The size gap between these two results is the cost of Conv layers staying in FP32.

Cross-framework compression comparison. Left: ResNet-18. TorchAO and ORT dynamic both achieve 1.03× (Linear/MatMul only); ORT static gives 3.95× by quantizing all Conv layers. Right: BERT-base. TorchAO gives 2.40× (nn.Linear only), while TFLite dynamic (3.94×), ORT dynamic (3.97×), and ORT static (3.97×) all converge near 4× because they quantize all weight tensors including embeddings. The compression difference between TorchAO and the other frameworks is format scope, not quantization quality.

Top: per-layer quantization sensitivity for ResNet-18. Bars extending right indicate layers that benefit from staying in FP32. The four most sensitive layers (conv 1, 2, 5, 15) are each the first convolution in a residual stage. Bottom: precision allocation map for four strategic configurations. Each row is one configuration; each column is a layer. Hatched cells are layers kept in FP32; solid cells are quantized to INT8. Accuracy and model size are annotated on the right.

Cross-framework numerical equivalence for ResNet-18 (top) and BERT-base (bottom). Bars show max |Δ logit| vs the PyTorch FP32 baseline on a log scale. ONNX FP32 export introduces negligible divergence (~10⁻⁵). Quantization shifts logits by 10⁻¹ to 10⁰, but prediction agreement remains above 99% for all variants. The largest divergence is ORT Dynamic INT8 on BERT — cosine similarity 0.9736 — where the narrow 2-class output space amplifies quantization error.

Logit-level agreement: FP32 baseline vs quantized variant. Left: ResNet-18 vs ORT Static INT8 (1000 logits × 100 samples). Points cluster tightly along the y = x identity line across the full [-10, +30] range despite 100% operator coverage. Right: BERT-base vs ORT Dynamic INT8 (2 logits × 100 samples). Visible scatter away from the diagonal, especially for negative logits around -4, with several points drifting by more than 1 unit. The narrow output range makes the same absolute error proportionally larger.

Operator quantization coverage by ONNX variant. Left: ResNet-18 ORT Dynamic — 4% coverage, only 1 MatMulInteger node (the classification head's Gemm) out of 51 compute ops. All 20 Conv and 17 Relu stay in FP32. Center: ResNet-18 ORT Static — 100% coverage, all 20 Conv, 8 Add, and the Gemm wrapped in QDQ pairs. Right: BERT-base ORT Dynamic — 28% coverage, 74 MatMulInteger nodes (the linear projections) quantized, but LayerNormalization, Softmax, Div, and Sqrt remain in FP32.

• Quantization workflows diverge across frameworks in default semantics, operator fusion rules, and calibration implementations. Parity requires tensor-level validation, not just accuracy checks.
• TorchAO's quantize_() targets nn.Linear only—compressing BERT-base by 2.40× and TinyLlama by 3.38×, but leaving Conv2d-dominated ResNet-18 at 1.03×.
• TFLite's converter handles quantization, optimization, and serialization in one pass. Dynamic range achieves 3.68× on MobileNetV2 with no calibration data.
• ONNX export is essentially lossless (max logit divergence ~10⁻⁵). Larger divergence in quantized variants comes from quantization arithmetic, not the export.
• ORT dynamic quantization targets MatMul/Gemm, effective for transformers (3.97× on BERT) but not CNNs (1.03× on ResNet-18). ORT static with QDQ format quantizes all operators (3.95× on ResNet-18).
• Accuracy preservation does not guarantee numerical equivalence. BERT under ORT Dynamic showed one prediction flip in 100 samples despite maintaining 96% overall accuracy.
• Operator coverage determines speedup—low op-count coverage can still be effective if those operators dominate compute.
• When validation fails, compare ONNX FP32 against the PyTorch baseline first to isolate export error from quantization error.

FAQ