9 Shipping with the Right Toolchain

Hardware selects the runtime family, and the execution provider selects which kernels dispatch. The leaf node is the production artifact. The two routes to TensorRT and OpenVINO, standalone versus via ORT, trade peak latency against a unified serving surface across heterogeneous fleets. Branches marked Chapter 10 and Chapter 11 point to regimes covered there.

BERT-base SST-2 on CPU EP. Dynamic INT8 is 1.35× faster than static INT8 at the same accuracy band. Newcomers find this direction counterintuitive; ORT documents it as expected behavior for transformer-class graphs.

Transformer-aware optimizer on BERT-base, CUDA EP, batch 8, sequence 128. The 1.17× reflects fusion alone. ORT_ENABLE_ALL at session-init does not run these transformer-specific patterns.

ResNet-18 static INT8, three calibration methods on CPU EP. Top-1 agreement collapses to a sub-1 pp band. The Entropy-latency spike is most likely a single-run measurement artifact; the three graphs have identical op types and node counts.

SessionOptions and io_binding on the FP32-optimized BERT graph, CUDA EP. All three configurations within 30 µs of each other. Same levers pay off differently on raw exports and on per-request LLM serving; this graph is already past the regime where they matter.

ResNet-18 inference latency on L4, batch 32. Implicit INT8 reaches 1.04 ms (6.44× over FP32) and outperforms explicit INT8's 1.73 ms (3.84×). The direction is counterintuitive at first, explicit is the recommended path and the engine inspector explains it.

BERT-base SST-2 latency on L4, batch 8, seq 128. Explicit INT8 leads at 2.73 ms; implicit INT8 lands within measurement noise of FP16. Dotted lines mark Table 9.1's CUDA EP INT8 (14.4 ms) and FP32-optimized (9 ms) references. TRT's INT8 explicit closes the gap section 9.2 identified.

ResNet-18 latency on c3-standard-4 (Intel Xeon Platinum 8481C, Sapphire Rapids), batch 32. OV native INT8 reaches 41.79 ms (9.55× over FP32) on AMX-INT8 tile registers; OV native BF16 reaches 58.04 ms (6.88×) on AMX-BF16. The via-ORT FP32 path costs essentially nothing (1.01×); via-ORT INT8 costs 38% relative to native. The asymmetry is informative. The two INT8 routes hand OpenVINO different artifacts.

BERT-base SST-2 latency on c3-standard-4, batch 8, seq 128. OV native BF16 (121.01 ms, 5.80×) and OV native INT8 (110.17 ms, 6.37×) both clear section 9.2's Cascade Lake CPU EP baseline by more than 2×. Via-ORT INT8 reaches 96.28 ms (7.29×), the fastest cell in the table, and the reason is recipe asymmetry, not runtime advantage.

Packaging verification matrix on an NVIDIA L4 host. Rows are artifact kinds; columns are runtimes. Solid cells executed at the claimed precision (the OpenVINO Core CPU column reports per-op precision fractions; ORT cells verify load and run, since ORT's Python API does not expose per-op precision). Hatched cells loaded but fell back. ResNet INT8 × ORT TRT EP routed 19 nodes to CUDA EP because TensorRT's parser rejects ORT's INT32 zero-point format on Conv DQ; BERT INT8 × ORT CUDA EP is the 9.2 anti-pattern resurfacing. Filled cells are runtime, hardware, or artifact absent; information about the deploy target, not bugs.

Packaging verification matrix on a c3-standard-4 host (Intel Xeon Platinum 8481C, Sapphire Rapids). The ORT/OV EP column lights up for the first time. The Section 9.4 via-ORT route runs end-to-end on AMX silicon. ORT/CUDA EP and ORT/TRT EP block (no GPU). The OV/CPU INT8 fractions match the L4 host within rounding (40% ResNet, 22-27% BERT); the BF16 fractions in the same column move sharply, from 0% on FP32 ONNX × OV/CPU on the L4 host to 26% on c3, because OpenVINO's INFERENCE_PRECISION_HINT defaults to BF16 on AMX-capable CPUs and auto-downcasts FP32 weights at compile time.

• Hardware selects the runtime family, the execution provider selects which kernels dispatch, and the artifact is whatever the winning runtime consumes: a TensorRT .plan, an OpenVINO IR pair, or a quantized ONNX served through ORT. When kernels for the target dtype are missing, runtimes fall back to FP32 silently, so verifying the executing dtype on the deploy host is the only proof the claimed precision is real.
• ORT's three quantization entry points are not interchangeable. On transformers, dynamic INT8 beats static because the static default quantizes the embedding lookup; restricting quantization to MatMul restores production behavior. Graph optimization that fuses Attention, LayerNorm, and GELU is a separate pass layered on top, and INT8 belongs on the CPU EP unless the TensorRT EP is in the provider list.
• TensorRT reaches INT8 through two paths that build different engines from the same model. The implicit calibrator path is fastest on simple CNNs but can silently demote a transformer to FP16 when key tensors never receive scales. The explicit QDQ-in-ONNX path carries a small reformat cost on CNNs but holds INT8 through the encoder on BERT.
• OpenVINO compiles two artifacts against the same CPU plugin, and the recipe, not the runtime, decides which wins. The native NNCF transformer preset favors Conv-heavy graphs; the leaner via-ORT recipe favors transformer matmuls. On Sapphire Rapids and later, BF16 captures most of the INT8 speedup without calibration and is the cheapest deliverable when the latency budget allows.
• Every artifact ships with a manifest sidecar and a content hash that pin the quantizer, format, calibration fingerprint, required hardware, and runtime compatibility. Precision lives in the artifact graph rather than the serving config, so a Triton INT8 config looks almost identical to its FP32 sibling. Run the verification matrix on the deploy host before opening the gate: move the artifacts to new silicon and the reachability columns shift while the INT8 fraction barely moves, because the recipe chose what to quantize.

FAQ