Overview
10 Running on CPUs and Using GGUF
The chapter explains how CPU-first LLM inference can be a rational deployment choice, but only when it is judged by cost, workload shape, and operational fit rather than raw speed alone. Because decode on CPUs is bandwidth-bound, the way weights are stored and accessed matters more than FLOPS, and small mistakes in quantization, build flags, or thread placement can quietly cut throughput by 10–30% without any obvious error. The discussion frames CPU inference as attractive for fleet bin-packing, unavailable GPU inventory, bursty traffic, large-model headroom, or cases where warm CPU capacity is already in place, while warning that GPUs remain structurally cheaper per token when they can be kept busy.
The text then unpacks GGUF as the file format that makes this CPU path practical. GGUF stores model metadata, tokenizer information, tensor descriptors, and packed tensor data in one self-describing file that can be memory-mapped directly, so the file on disk effectively becomes the in-memory weight store. It distinguishes legacy quants, k-quants, and IQ-quants: legacy formats use one scale per block, k-quants add hierarchical super-block and sub-scale structure to preserve quality at lower bit widths, and IQ-quants use codebooks guided by an importance matrix to push compression even further. A representative GGUF such as Q4_K_M is shown to be a recipe rather than a uniform file-wide quantization, with most tensors at the base format and sensitive ones promoted to higher precision.
Finally, the chapter describes the end-to-end conversion and deployment pipeline, from Hugging Face safetensors to F16 GGUF, then to quantized GGUF variants, and it evaluates those variants on size, perplexity, fluency, decode speed, kernel family, memory use, and thread count. The results favor IQ4_XS on Apple Silicon and modern x86 in throughput and compression, while Q4_K_M remains the safer portable default on less capable hardware. On Sapphire Rapids, AVX-512 with VNNI is the best decode build, while AMX hurts batch-1 generation even though it helps prompt processing; the chapter also shows that runtime memory is not just the on-disk file size, since the KV cache and compute buffer can make a model much larger in RAM, and that cloud-vCPU systems may benefit from using all available threads rather than stopping at physical cores. The overarching lesson is that production CPU inference succeeds when file format, kernel choice, and provisioning are tuned to the actual workload.
Effective cost per million Llama-2-7B Q4_K_M tokens versus sustained demand, on Google Cloud c3-standard-22 (CPU-only, solid) and g2-standard-4 with one NVIDIA L4 (dashed) in us-central1 at on-demand list pricing. Each curve declines hyperbolically until the instance's single-stream throughput ceiling (CPU at 20 tok/s, GPU at 50 tok/s; tick marks at successive replica boundaries), then steps up as additional replicas absorb demand above ceiling. Below saturation the ratio between curves equals the ratio of hourly prices (1.57×). At the GPU's saturation the ratio is roughly 3.9×, widening further at higher demand because the CPU instance saturates at 2.5× lower throughput. The L4 wins on cost per token at every demand point in the plot; the gap is bounded between 1.57× and roughly 5×.
GGUF quantization formats by bits per weight, including per-block scale and metadata overhead. Legacy quants (dotted) span Q4_0 (4.50 bpw) through Q8_0 (8.50). K-quants (diagonal) span Q2_K (2.62) through Q6_K (6.56). Bars show each base format's bpw; the recipe-level M-variants (Q3_K_M, Q4_K_M, Q5_K_M) sit 0.2–0.4 bpw above their base because of bumped tensors, as 10.2.4 makes explicit. IQ-quants (cross) span IQ1_S (1.56) through IQ4_XS (4.25). Importance-matrix calibration pushes below the k-quant floor. Reference dashes mark INT4 and INT8.
GGUF file layout: llama-2-7b.Q4_K_M.gguf (3.80 GB total). Front-of-file detail (logarithmic byte axis) shows the 17-byte alignment pad between tensor info and tensor data. Hex dump highlights the four header fields , magic GGUF, version 2, tensor count 291, metadata count 19, followed by the start of the first KV (general.architecture).
The HF-to-GGUF conversion pipeline: top row is the spine: HF safetensors → convert_hf_to_gguf.py (PyTorch) → F16 GGUF → llama-quantize → quantized output. Bottom row is the IQ-quant detour: F16 GGUF → llama-imatrix → imatrix.gguf, with the dashed arrow marking the imatrix as a hard prerequisite for IQ4_XS only; Q8_0 and the k-quants ignore it.
Wikitext-2 perplexity (4 chunks × 512 tokens, CPU-only) versus on-disk size. F16 (12.85 GB, PPL 5.27) anchors the lossless baseline. F16 → Q8_0 → Q5_K_M is a clean staircase down. Q4_K_M sits to the upper side of Q5_K_M as expected. IQ4_XS sits to the lower-left of Q4_K_M—smaller and lower perplexity at the same bit tier.
The first 64 tokens generated by each variant from the prompt "The Apollo program was the third United States human spaceflight program carried out by", at --temp 0 --seed 42 on Apple M3 Metal. F16 and Q8_0 are byte-identical for these 64 tokens. Q5_K_M, Q4_K_M, and IQ4_XS share the first 12 tokens with the baseline, then diverge stylistically. None collapses, repeats, or degrades.
Decode time per token on Apple M3/Metal for the same 64-token completion. F16 decodes at 6.5 tok/s; IQ4_XS at 23.4 tok/s — a 3.6× speedup. Decode time scales nearly linearly with bytes-read-per-token because 7 B-parameter decode is memory-bandwidth-bound on this hardware. The italic footnote on the figure flags that this ranking is for Apple Silicon + Metal. On older x86 CPUs, the IQ4_XS ↔ Q4_K_M gap can shrink or invert.
Vector lane progression for one Q4_K block decode on AVX-512 with VNNI. The 256-weight superblock from Section 10.2 traverses three register states: a single 64-byte cache-line load fills zmm0 with packed nibbles, two unpack instructions split low and high nibbles into 64 INT8 lanes in zmm1, and VPDPBUSD accumulates 16 INT32 dot products in zmm2 that horizontal-reduce to a scalar. The NEON path on Apple Silicon executes the same sequence at one-quarter the lane count per instruction, with SDOT (Armv8.4-A) as the dot-product equivalent of VPDPBUSD.
Decode throughput on Llama-2-7B across five quantization formats and three llama.cpp builds. F16 is flat across all three builds because it does not enter any of the integer dot-product instructions the build flags govern; its column measures the host's memory bandwidth alone. The integer formats show single-digit-percent gains from AVX-512+VNNI over AVX-2 and consistent regressions on the AMX-enabled build. Q5_K_M's regression on AMX is roughly three times the regressions Q4_K_M and Q8_0 seen on the same build.
IQ4_XS decode throughput as a multiple of Q4_K_M decode throughput on the same hardware, same model. Ratios above 1.0 mean IQ4_XS wins despite being the more complex format; ratios below 1.0 mean codebook lookups serialize faster than the bandwidth saving recovers. Measured: avx2 1.13, avx512 1.13, avx512-amx 1.06. The avx2 column is the kernel family Section 10.3 worried about, and the lead does not invert there.
Runtime memory footprint for Q4_K_M Llama-2-7B (MHA, 32 KV heads) and Q4_K_M Llama-3-8B (GQA, 8 KV heads) at 2048 and 4096 context. Weights are nearly fixed across context lengths; the KV cache (/// hatching) scales linearly with context and dominates the footprint at long ctx for the MHA model. The compute buffer (xxx hatching referred as comp in the fig), activation scratch the runtime allocates per forward pass, sits around 100 MiB for Llama-2-7B and grows to 266 MiB for Llama-3-8B because activation scratch on the output side scales with vocabulary, and Llama-3's 128K vocabulary is 4× Llama-2's. Llama-3-8B's 4× reduction in KV heads (GQA) cuts the KV cache footprint by 4× at the same context: the Llama-3-8B 4096-ctx bar's KV segment is 512 MiB versus 1024 MiB for the Llama-2-7B 2048-ctx bar despite holding twice as many tokens of state. Numbers measured from llama-cli stderr on c3-standard-22, llama.cpp build at commit 6d57a49.
Decode throughput on c3-standard-22 (Sapphire Rapids, 11 physical cores × 2 SMT) versus llama.cpp -t thread count, AVX-512+VNNI build, Q4_K_M Llama-2-7B, n=64 token generation, mean over 5 repetitions. Throughput climbs through the full vCPU count rather than peaking at the physical-core count: 16.62 tok/s at t=11 lifts to 21.02 tok/s at t=22, a 26% gain that the kernel extracts by recruiting SMT siblings. A local minimum at t=12 (15.81 tok/s, 4.9% below t=11) marks the first SMT-sibling activation under L1/L2 contention; throughput recovers past t=14 once enough cores share that pattern. The dashed line at t=11 marks the physical core count; the dashed line at t=22 marks the full vCPU count.
Summary
- The CPU win is cost consistency and scale-out, not a lower peak price. At full utilization a GPU is cheaper per token, and the L4-versus-c3 matchup in 10.1 put that gap near 4×, bounded rather than the order-of-magnitude folklore sometimes claims. What CPU buys is steadiness. CPU instances bill per core-hour and scale out linearly, so per-token cost is bounded by the batch-1 figure and tracks load smoothly, whereas a GPU bills for the whole accelerator whether or not enough concurrent requests arrive to keep it saturated. For bursty, latency-bound workloads that never reach that saturation, predictable CPU cost beats throughput you cannot keep busy, and CPU is the only path at all when the deployment target has no GPU.
- GGUF is a self-contained binary container, and its quantization formats are a separate ecosystem from Chapter 9's runtimes. A single file carries weights, scales, tokenizer, chat template, and metadata, with no schema dependency on PyTorch or transformers at load time. K-quants such as Q4_K_M pack weights into 256-element superblocks with one master scale and a sub-scale per 16-weight chunk, while i-quants such as IQ4_XS replace the scale with a non-uniform codebook chosen by importance-matrix calibration. Neither layout round-trips through ONNX, which has no operator for the superblock arithmetic or the codebook lookup. And unlike ONNX's QDQ format, GGUF is fail-closed: each header code maps to exactly one decoder, so the runtime either loads a file at its on-disk format or refuses it outright.
- The conversion pipeline is one Python script and two C++ binaries. convert_hf_to_gguf.py rewrites a Hugging Face checkpoint into an FP16 GGUF, llama-quantize rewrites that into a target format such as Q4_K_M or IQ4_XS, and the IQ path additionally needs an importance matrix from llama-imatrix before its codebook fit.
- Decode is bandwidth-bound, so the format is the large lever and the kernel family is a small correction. A 7B model reads its full weight footprint per generated token, which caps throughput at the rate DRAM supplies bytes, a ceiling no vector width or fused-dot instruction can cross. Across the quant ladder on one Sapphire Rapids build, throughput moved about 2.9× from F16 to IQ4_XS; across the three kernel families on a fixed format it moved only single-digit percentages. Compiling AMX in was a 6–29% regression for batch-1 decode, because AMX is a batched-matmul unit with no work to amortize at batch 1. The shipping default is therefore AVX-512 with VNNI and AMX off, with IQ4_XS the throughput-optimal format on every CPU kernel family measured and Q4_K_M the safe fallback wherever a runtime cannot read codebooks.
- Runtime memory is three terms, and on cloud silicon the obvious thread count is wrong. On-disk size accounts only for the weights; the KV cache, which is closed-form at 2 × n_layers × n_kv_heads × head_dim × ctx × 2 bytes, catches and then overtakes it at long context, while the compute buffer scales with vocabulary more than with parameter count. GQA cuts the KV cache by the KV-head ratio, which is why an 8-KV-head Llama-3-8B holds twice the tokens of a 32-KV-head Llama-2-7B in a quarter of the cache. For thread count, the published "one thread per physical core" rule held on bare metal but not on a shared-tenant c3-standard-22, where neighbor noise leaves bandwidth headroom that SMT recruits, so -t $(nproc) beat the physical-core count by 26%. Provision against the runtime total, and sweep -t once per model, format, and host before committing.
Quantization and Fast Inference ebook for free
