4 Running Inference
This chapter introduces the practical concerns involved in running inference with small and large language models, using GPT-Neo as a representative Transformer-based example. It explains how models and tokenizers are loaded with Hugging Face Transformers, how device placement can be handled automatically with Accelerate, and why memory requirements depend heavily on model size and numeric precision. It also shows that the same generation API can support text completion, batched prompts, few-shot classification-style prompting, and even code generation, while emphasizing that pretrained open-source models often need fine-tuning, LoRA, or retrieval augmentation before production use.
The chapter then focuses on evaluating and estimating the cost of inference. It describes the need to assess generated content before deployment and introduces evaluation frameworks that can test models across standardized or custom tasks using metrics such as perplexity. For infrastructure planning, it explains how to estimate GPU memory needs from the number of model parameters, precision format, and expected overhead, then connects those requirements to production metrics such as Time To First Token, Time Per Output Token, latency, throughput, and GPU utilization. These measures help determine whether a model can run efficiently on available hardware and how much serving it will cost.
The final part examines ways to improve performance and reduce cost. It explains how GPU compute bandwidth, memory capacity, and memory bandwidth affect inference, and how to determine whether a workload is memory-bound or compute-bound. It highlights the importance of the key-value cache for faster autoregressive generation, discusses batching as a way to improve throughput when latency constraints allow it, and shows how generation time can be estimated from model size, memory bandwidth, and prefill cost. The chapter also introduces broader optimization strategies such as operator fusion, quantization, compression, parallelization, and DeepSpeed, showing that optimized inference kernels and model parallelism can reduce latency, though results should always be validated for quality and accuracy.
The (Q * KT) * V computation process with caching
Summary
- GPT-Neo 2.7B model demonstrates text completion, few-shot learning, and code generation capabilities using the Transformers library
- Memory requirements calculated with formula: memory (GB) = (4*P)/(32÷F)*1.2, where P is parameters in billions and F is precision bits
- GPT-Neo 2.7B requires 12.96 GB in fp32 or 5.4 GB in fp16 precision; Mistral 7B needs 33.6 GB in fp32 or 16.2 GB in fp16
- Key inference metrics include Time To First Token (TTFT), Time Per Output Token (TPOT), latency, and throughput
- KV cache stores attention layer states to avoid recalculating for each token, providing 5x speed improvement in testing
- Ops:byte ratio determines if system is memory-bound or compute-bound; T4 GPU has 216.6 ops/byte ratio
- Arithmetic intensity measures compute operations per byte accessed; GPT-Neo's 30.29 ops/byte indicates memory-bounded performance
- Dynamic batching processes multiple requests simultaneously, improving throughput by reusing model weights already in GPU memory
- Batch size limited by available GPU memory after accounting for model weights and KV cache
- Token generation time estimated by dividing bytes moved by GPU memory bandwidth; GPT-Neo generates roughly one token per 36 milliseconds on T4 GPU
- LM-Eval framework provides standardized evaluation with 200+ built-in tasks measuring metrics like word perplexity, byte perplexity, and bits per byte
- DeepSpeed library optimizes inference through model parallelism, kernel injection, and ZeRO optimizer implementation
- DeepSpeed achieved 18% latency reduction for GPT-Neo inference compared to vanilla implementation
- Accelerate library provides estimate-memory command for calculating precise memory requirements across precision levels
FAQ
What is the main goal of chapter 4, “Running Inference”?
The chapter introduces the SLM/LLM inference space. It explains how to generate different kinds of content, how to estimate inference memory and cost requirements, and where to look for performance improvements and cost savings when serving models in production.Which reference model is used in the chapter for inference examples?
The chapter uses GPT-Neo 2.7B from EleutherAI as the reference open source model. It has 2.7 billion trainable parameters and is available on Hugging Face atEleutherAI/gpt-neo-2.7B. Although GPT-Neo is no longer actively developed, it remains useful for explaining Transformer-based inference concepts.How can GPT-Neo be loaded for text generation with Hugging Face Transformers?
The model can be loaded usingGPTNeoForCausalLM and its tokenizer using GPT2Tokenizer. The chapter uses device_map="auto" so Hugging Face Accelerate can automatically place model layers on the fastest available device, such as a GPU, and offload parts to CPU or disk if needed.What does device_map="auto" do during model loading?
device_map="auto" lets the Accelerate library decide where each layer of the model should be placed. It tries to maximize the use of available GPU memory and can offload layers to CPU RAM or disk if the model does not fully fit on the GPU. This is transparent to the developer; the inference code remains the same.How can you estimate how much memory a Hugging Face model needs?
You can use the Accelerate command-line tool:accelerate estimate-memory MODEL_ID. For example, accelerate estimate-memory EleutherAI/gpt-neo-2.7B reports estimated memory usage for different precisions such as float32, float16, int8, and int4.What kinds of content generation are demonstrated in the chapter?
The chapter demonstrates several inference use cases:- Text completion, where the model continues a prompt.
- Batch text generation, where multiple prompts are completed together.
- Few-shot learning, where examples in the prompt guide the model to produce similar outputs, such as sentiment labels.
- Code generation, where the model generates code from an instruction-style prompt.
Why should generated code from a general LLM be reviewed by a developer?
Generated code may be incomplete, poorly formatted, inefficient, or incorrect. The chapter notes that GPT-Neo can generate code because it was trained on data that includes GitHub repositories, but its output still needs developer review, linting, and possible improvement. Specialized code models may produce better results.What is LM-Eval and why is it useful?
LM-Eval, or EleutherAI’s Language Model Evaluation Harness, is an open source Python framework for evaluating LLMs across many tasks. It supports more than 200 built-in evaluation tasks, custom tasks, Hugging Face models, OpenAI API models, and some quantized model formats. It helps measure model quality before deciding whether to deploy, mitigate, tune, or optimize a model.How is inference memory roughly calculated for a model?
The chapter gives a formula based on the number of model parameters and precision: memory depends on the bytes per parameter, the total number of trainable parameters, the selected precision, and an additional overhead factor of about 20%. For example, GPT-Neo 2.7B loaded infp32 is estimated at about 12.96 GB including overhead, while fp16 requires much less memory.What are TTFT, TPOT, latency, and throughput in LLM inference?
- TTFT, or Time To First Token, is how quickly a user sees the first output token after submitting a prompt.
- TPOT, or Time Per Output Token, is the time required to generate each output token for each active user.
- Latency is the overall time needed to generate a full response.
- Throughput is the number of output tokens per second the serving system can generate across users and requests.
Domain-Specific Small Language Models ebook for free
