11 Targeting Edge and Mobile Devices

The chapter reframes edge deployment as a budgeting problem rather than a pure model-optimization problem. On phones and similar devices, the key constraints are user-visible latency, sustained thermal behavior, and power draw, all of which must be satisfied together before a configuration is worth considering. It contrasts this with server deployments, where the main concern is cost efficiency, and shows that the same model can behave very differently depending on whether it runs on a laptop, a phone, an SBC, or a small-box accelerator.

Across Android and Apple platforms, the chapter emphasizes that runtime routing matters as much as quantization choice. On Android, TensorFlow Lite and LiteRT-LM expose different conversion and backend surfaces, and only fully integer graphs can take advantage of accelerator paths like NNAPI or vendor delegates; partial quantization or poor calibration can produce large accuracy cliffs or silent fallbacks. On Apple silicon, Core ML’s compute-unit hints and MLX’s transformer-focused runtime determine whether workloads reach the Neural Engine or spill onto the GPU, with op-support gaps making some transformer workloads GPU-bound regardless of weight format.

The final lesson is that deployment success depends on measuring the whole pipeline, not just the forward pass. Preprocessing and postprocessing can dominate total latency, especially when platform-native primitives are missing or bypassed, as seen in Android audio feature extraction and Python-based vision pipelines. The chapter closes by recommending native platform tools, end-to-end latency reporting, and tier selection based on the actual deployment surface, since the right runtime, accelerator, and preprocessing strategy are all inseparable parts of shipping efficient edge inference.

Cross-modal design space on MacBook Air M3 (laptop) and Google Pixel 10 Pro (phone). Marker shape encodes modality (circle = vision, triangle = audio, square = LLM). Twelve points populate the chart. Five M3 vision variants sit at 4.4–6.6 W, two M3 audio variants at 5.7–6.7 W, Pixel vision at 1.75 W, Pixel audio at 3.33 W, three Pixel LLM points at 5.51 W. The iPhone 16 is absent. iOS exposes no third-party API for instantaneous power (MetricKit reports daily-aggregated CPU and GPU time, not a milliwatt figure at the cadence this axis needs). M3 LLM is also absent. MLX publishes neither TTFT nor power telemetry, so the cell cannot land on either axis. Latency is p50 for vision and audio; TTFT for LLM. Power is powermetrics on macOS and BatteryManager on Android, both over a 30 s window.

Thermal retention across modalities and devices. Bar height is 100 × last_30s / first_30s of sustained throughput; 100% (dashed line) is the no-throttle baseline. Bar hatches encodes modality (vision plain, audio dotted, LLM cross-hatch). Annotations report retention percentage and first → last absolute throughput in the modality's native unit. The M3 vision 60 s bar loses 35.6%; the M3 audio 300 s bar shows a 54.1% gain; the Pixel bars hold within 1.3 percentage points of 100%.

Budget-feasible region for EfficientNet-Lite0 at p95 ≤ 33 ms and mean power ≤ 5 W. The diagonal-striped region is the feasible envelope. Inside are the Pixel INT8 XNNPACK point and the two M3 Core ML INT8 points. Outside on the power axis, every M3 FP16 and palettize variant lands at 5.9–6.6 W. No configuration fails on latency. EfficientNet-Lite0 clears 33 ms everywhere measured.

Cross-modal latency on shipped devices, canonical per-modality configuration, log y-axis. Vision column shows M3 4-bit palettize at 0.36 ms, iPhone 4-bit palettize at 0.42 ms, Pixel INT8 XNNPACK at 4.09 ms. Audio column shows M3 Whisper-tiny palettize-6-bit at 13.5 ms, iPhone INT8-linear at 17.9 ms, Pixel dynamic-range NNAPI at 290.5 ms. LLM column shows iPhone Core ML prefill1_int4 at 17 ms per call, Pixel LiteRT-LM INT8 at TTFT 428 ms for a 32-token prompt. The M3 LLM cell is empty. MLX exposes only steady-state per-token latency, not TTFT.

EfficientNet-Lite0 size × ImageNet top-1 accuracy across the four TFLite converter paths, measured on a 1,000-image ImageNet val subset. FP32 at 18.55 MB lands at 66.8% top-1 (87.7% top-5), the unquantized baseline. Dynamic-range at 4.88 MB lands at 66.5% top-1 (87.9% top-5). That is a 3.8× file-size reduction with essentially no accuracy loss. INT8 at 5.14 MB lands at 49.0% top-1 (86.8% top-5); INT16×8 at 5.40 MB lands at 50.6% top-1 (87.4% top-5). The 17-percentage-point top-1 cliff on the two activation-quantized paths is a calibration artifact, not a property of the integer kernels. The command ch11_2_tflite.py convert ran with a synthetic uniform-random representative dataset because data/calib/ was empty at conversion time. The top-5 numbers (within one percentage point of FP32 on all four variants) confirm the cliff is calibration-shape, not capacity loss; a proper ImageNet calibration shard recovers the gap.

TFLite per-tensor dtype map across the four converter paths. Each bar is the tensor count per dtype in the produced flatbuffer. FP32 has 161 float32 tensors plus one int32 (a shape constant). Dynamic-range has 60 int8 (weights), 129 float32 (activations and biases), 15 int32. The converter has quantized only the weights and left every activation tensor in FP32. INT8 has 111 int8, 51 int32, zero float32. The integer pipeline is end-to-end, no float remains in the graph. INT16×8 has 62 int16 activations, 50 int8 weights, 50 int64 (zero-points and scales held at higher precision), 4 int32, and two float32 tensors, the input and output. The two float fallbacks are by design (the operator at the I/O boundary always materializes its tensor in the inference-input dtype the API was constructed with), but they are also why the experimental supported-ops set is called experimental.

Pixel 10 Pro runtime matrix across vision (EfficientNet TFLite), audio (Whisper-tiny TFLite encoder), and language (Llama-3.2-1B LiteRT-LM), shared log y-axis. Vision panel shows four TFLite variants × four delegates (XNNPACK one- and four-thread, GPU over the Tensor G5's Imagination PowerVR DXT-48-1536, NNAPI). NNAPI rejects INT8 and INT16×8 at compile time and the cells appear empty. INT16×8 is a uniform ~143 ms across delegates because XNNPACK has no fused kernels for the supported-ops set and every cell falls back to the reference interpreter. Audio panel shows two TFLite variants (FP32 and dynamic-range, since Whisper-tiny is not calibrated on this fleet because no representative mel-spectrogram dataset is in scope) × three delegates. The XNNPACK 1T column is absent because the AWS Device Farm sweep collected only XNNPACK 4T / NNAPI / GPU for the audio matrix. LLM panel shows LiteRT-LM's single INT8 bundle, three prompt lengths (32 / 256 / 1024 tokens), TTFT only. LiteRT-LM does not expose a variant axis at the application API surface, every cell ran on the GPU.

EfficientNet-Lite0 p50 latency on Core ML, five variants (fp16, int8_linear, int8_weight_only, palettize_4bit, palettize_6bit) across four M3 compute-unit hints plus iPhone 16 all (the only mode Xcode Performance Reports expose). The cpuAndNeuralEngine and all columns on M3 land roughly 6× faster than the cpuOnly column on fp16 and roughly 7× faster on int8_linear. Inside the cpuAndNeuralEngine column the five variants collapse to within 0.1 ms of each other, because the Neural Engine quantizes activations internally and reads any compatible weight format at the same throughput.

Apple Neural Engine op coverage on the iPhone 16 by modality, sourced from .mlperfreport files via the MLComputePlan API. EfficientNet-Lite0 (five variants) and Whisper-tiny encoder (five variants) place every compute op on the ANE. The four Llama-3.2-1B variants (prefill128_int4_per_block, prefill128_int8_per_block, prefill1_int4_per_block, prefill1_int8_per_block) place zero ops on the ANE and route entirely to the GPU.

EfficientNet-Lite0 mean latency on the MacBook Air M3 across two runtimes. Core ML (coreml_fp16, default all compute units resolving to the ANE) at 1.5 ms, and PyTorch MPS (mps_fp16) at 45.5 ms. The MLX cell is empty. MLX ships no vision kernels in its primary scope and is structured as Core ML's transformer sibling, not a vision replacement.

Llama-3.2-1B autoregressive decode throughput, three M3 MLX variants × three prompt lengths plus Pixel 10 Pro LiteRT-LM INT8 baseline. M3 mlx_q4_g128 peaks at 78.5 tok/s at 32-token prompt and falls to 50.4 tok/s at 1024 tokens. M3 mlx_q8_g128 reads 54.6 / 48.7 / 36.2 tok/s. M3 mlx_fp16 reads 29.5 / 29.7 / 25.2 tok/s. Pixel 10 Pro LiteRT-LM INT8 reads 23.8 / 24.9 / 25.7 tok/s, flat across prompt lengths because the Tensor G5's PowerVR DXT-48-1536 GPU and its memory subsystem hit a bandwidth ceiling at every length. iPhone 16 Core ML Llama cell empty, prefill-only conversion path.

Pre/post breakdown by modality and device, log y-axis. M3 vision (Python PIL or OpenCV) reads ~30 ms decode + ~17 ms resize + ~3 ms normalize, totalling ~50 ms, already above the 33 ms frame budget for a 30 fps camera before inference begins. M3 audio reads ~11 ms decode + ~22 ms log-mel via Apple Accelerate.vDSP, totalling ~33 ms. M3 text reads ~0.8 ms tokenize + ~0.1 ms detokenize via Hugging Face Rust-backed tokenizers for a 128-token Llama prompt. Pixel vision reads ~0.4 ms decode + ~0.1 ms resize + ~4.7 ms normalize via BitmapFactory + Canvas + Kotlin normalize loop, totalling ~5.3 ms, faster than every M3 vision pipeline measured. Pixel audio reads ~243 ms log-mel via a Kotlin radix-2 STFT implementation, two orders of magnitude slower than M3 Accelerate.vDSP on the same Whisper-tiny feature pipeline.

• The edge regime swaps both axes that drove chapters 9 and 10, and a budget is the filter that survives the swap. Cost becomes joules per inference, and the failure mode becomes a thermal knee minutes into sustained load. The defense is a latency-and-power budget applied before optimization rather than after, two axes that must hold jointly. Those are p95 latency on the user-visible path and mean power over a sustained window. The binding axis is set by the workload, vision on power, ASR on energy per inference, and LLMs on prefill TTFT once prompts pass a few dozen tokens.
• Cold-start latency and sustained throughput are different numbers, and the measurement source is part of the number. The M3 Neural Engine sheds about a third of its vision throughput over a 60-second sweep as skin-temperature throttling pulls clocks down, while the Pixel holds flat, so shipping software commits to the worse number. Power telemetry is just as uneven. The powermetrics reading is an estimate, BatteryManager is good to within 10–20%, and iOS exposes no instantaneous figure at a usable cadence.
• Android routes two runtimes over one flatbuffer family, and the production risk lives at the variant-by-delegate intersection. TFLite exposes four integer paths, and get_tensor_details() is the only check that names what the converter actually produced, since dynamic-range leaves activations in FP32 and NNAPI rejects integer vision graphs on the Tensor G5. LiteRT-LM is the transformer sibling that exposes prefill TTFT and per-token decode separately, which is why an LLM SLA reads in TTFT, and the compute_units recorded at measurement time is the only honest reading of where the work ran.
• On Apple silicon the compute-unit hint outranks the weight-variant choice, and it is a ceiling the runtime can silently override. cpuAndNeuralEngine runs EfficientNet-Lite0 six to seven times faster than cpuOnly, and inside that column the five weight variants collapse to within a tenth of a millisecond. The override fires against the Neural Engine's fixed, vision-era op table. Llama-3.2-1B lands at zero ANE coverage on every weight format, and palettized weights cost roughly twice their linear-quantized equivalents on small encoders. MLX is the M-series-only transformer sibling, not a vision replacement.
• Tier selection sits above runtime choice and is set by the deployment surface, not the model. A consumer phone routes to the phone tier, TFLite and LiteRT-LM or Core ML and MLX. A fixed 5–25 W installation routes to the SBC tier, where XNNPACK on the ARM CPU is the workhorse and an optional Coral or Hailo accelerator brings its own narrower op table. A throughput-bound, around-the-clock workload routes to the small-box tier and a vendor SDK, accepting lock-in and one to three points of calibration drift on the same source graph.
• Inference-only latency is a diagnostic; the latency the user feels is end-to-end, and pre/post can dominate the inference it surrounds. The Pixel's portable Kotlin log-mel STFT runs longer than the Whisper-tiny encoder it feeds, and a NEON-tuned FFT cuts more latency than any inference-side switch in 11.2. Report end-to-end p95 in any user-facing SLA, use the platform's native primitives where they exist, and where they do not, bake STFT-class ops into the graph or bind a tuned FFT through JNI.

FAQ