12 Delivering Proven Results with a Four-Bit LLM Server and an On-Device Vision Pipeline

The gated serving build: the pre-ship gate (left, diagonal hatch) is a GPU-free pytest suite that asserts the configuration and code are correct before deployment. The serving stack (center) runs the AWQ artifact on an Ada-class L4 and exports metrics over HTTP. The post-ship gate (right, cross-hatch) scrapes those metrics and sensor-grade GPU power into Prometheus, renders the Grafana dashboard, and runs a quality canary that re-sends a fixed prompt set greedily and compares the four-bit output against the full-precision reference.

Illustrative Grafana view of the live vLLM serving dashboard during the concurrency sweep: end-to-end request latency percentiles, prompt and generation token throughput, inter-token latency, and scheduler state. The screenshot shows the post-ship gate running and is not a quantitative source — the numbers in this section come from the metrics scraped into the results file.

Per-image stage breakdown for EfficientNet-Lite0 on the M3, stacked to the per-image mean of 6.95 ms. Decode and resize use diagonal hatching, normalize a cross-hatch and the Core ML forward pass a dotted fill. The dashed horizontal line marks the end-to-end p95 of 9.00 ms. The forward pass is 1.23 ms, 17.7% of the mean, while decode at 4.20 ms is roughly 60%; the stage the accelerator optimizes is the smallest segment, and decode owns the budget.

Illustrative Grafana view of the hand-rolled device dashboard: per-stage latency, end-to-end p95, estimated SoC power, op placement across the ANE, GPU, and CPU, and thermal retention. The panel titles carry the labels the metrics are stored under, including "estimated power (W) — powermetrics, not sensor-grade." The screenshot shows the post-ship view running and is not a quantitative source; the numbers in this section come from the device results file.

Peak generation throughput (left) and best time-to-first-token p95 (right) by format, derived from the sweep in Table 12.1. AWQ roughly doubles peak throughput (1,587 → 2,926 tokens/s) and cuts the best achievable TTFT p95 from 97 ms to 38 ms, a 2.6× reduction.

Equation 12.1

KV-cache occupancy (left) and request-queue depth (right) against offered concurrency on a log-2 axis, FP16 versus AWQ. FP16 is the solid line with circle markers, AWQ the dashed line with square markers. Occupancy stays below 2% and queue depth stays at zero across the entire sweep for both formats — the system never approaches its memory limit at this model size, so the saturation knee the freed memory is meant to push outward does not appear.

• A shipped quantized model clears two gates, and they cover different failures. The pre-ship gate is offline, deterministic, and GPU-free, and it fails the build when the artifact or its configuration is wrong before anything deploys. The post-ship gate is the live dashboard, watching for the drift the tests cannot predict. Both ask the same thing, whether the quantized model still behaves like full precision inside its latency, memory, and power budget.
• The four-bit LLM roughly doubled throughput for about half the energy per token, on the same card. Swapping FP16 for AWQ four-bit on one L4, with nothing else changed, raised peak throughput about 1.8x while GPU power stayed near 72 W, so energy per token fell from roughly 45 to 25 millijoules. The greedy canary put the four-bit endpoint at 89.9% token agreement with the FP16 reference, so the speed is paid for with a small measured quality cost, not a free one.
• Whether four-bit weights run as four bits is decided by the silicon, not just the artifact. AWQ needs a matching INT4 kernel, and vLLM has one only at compute capability sm80 and newer. The Ada-class L4 clears that floor. The same artifact on a Turing T4 falls back to FP16 and gives back everything quantization bought. Quantizing a model and serving it have different silicon floors.
• The vision pipeline cleared its sub-second budget with room to spare, and decode owned the budget, not inference. End-to-end p95 was 9 ms against a one-second target. The Core ML forward pass was 17.7% of the per-image cost while JPEG decode alone was about 60%, so the stage the accelerator was quantized to speed up is the smallest movable part of the path. A fast inference-only reading is a diagnostic worth tracking. The end-to-end number is what belongs in the target, on the device and on the server alike.
• Where the model actually ran is worth checking on both planes, because neither reports it by default. On the server the question is whether vLLM ran the four-bit kernel or upcast to FP16. On the device it is whether the Apple Neural Engine ran the model or the CPU did. Routing on Apple silicon is silent, compute_units=ALL is a permission rather than an instruction, and MLComputePlan is the only window onto the answer. It read 19% of operations on the ANE by count, which looks CPU-heavy, even though the 1.23 ms forward pass says the ANE carried the real arithmetic.
• The telemetry the server got for free was hand-rolled on the device. vLLM and DCGM publish Prometheus-shaped metrics over HTTP, so the server's dashboard cost a few lines of configuration. Core ML and MLX publish nothing of the sort, so the device pipeline ships its own exporter. Same questions, very different instrumentation cost.
• The limits you state next to a result are what make it trustworthy. The 3B model left so much KV headroom on a 24 GB card that occupancy stayed below 2% and the queue never filled, so the saturation knee freed memory is meant to push outward is a measured zero, not a missing measurement. It returns with the 7B or 8B model this build usually targets. Server power is sensor-grade and device power is estimated, two different classes of measurement, which is why the cross-plane energy figure was left unbuilt instead of drawn on a misleading shared axis. A result is only as trustworthy as the limits recorded next to it.

FAQ