11 Deployment and Serving
This chapter surveys practical ways to deploy and serve customized small language models across common target environments, from GPU-backed servers to local and edge devices. Its main theme is that serving choices depend heavily on the deployment context: high-throughput HTTP inference, lightweight custom APIs, cross-platform compiled runtimes, or fully on-device execution. Across all options, the chapter emphasizes ease of integration, performance benchmarking, model optimization, and careful matching of model size and precision to available hardware.
For server-side deployment, the chapter focuses on vLLM and FastAPI. vLLM is presented as a purpose-built inference and serving framework for language models, centered on PagedAttention, efficient KV-cache management, continuous batching, quantization support, streaming, distributed inference, and OpenAI-compatible APIs. It can be used for offline generation and chat workloads, including quantized GGUF models, or launched as an online HTTP server that client applications can call through OpenAI-style interfaces. The chapter also stresses the importance of post-deployment benchmarking, using metrics such as throughput, time to first token, time per output token, and inter-token latency. FastAPI is then introduced as a flexible Python framework for exposing model inference behind custom REST endpoints. It integrates naturally with Hugging Face pipelines, Transformers, ONNX Runtime, Pydantic data validation, asynchronous route handlers, and containerized deployment. The chapter shows that benchmarking different model formats, such as PyTorch, ONNX, optimized ONNX, and FP16 variants, helps select the best candidate before exposing it through a production API.
For edge and on-device scenarios, the chapter introduces MLC LLM, Android deployment options, mllm, and direct Hugging Face Transformers execution on Android. MLC LLM is described as a cross-platform compiler and inference engine that converts, quantizes, configures, compiles, and serves models through Python, REST, mobile, and other runtime targets while retaining an OpenAI-like API. On Android, MLC LLM can be used to build a native app and bundle model weights, although the workflow requires several non-Python development dependencies. The mllm framework offers another mobile-oriented path, using a lightweight C++ inference engine with support for converted and quantized models. Finally, the chapter describes a fully Python-based Android approach using Termux, Python virtual environments, PyTorch, and Transformers, while warning that local inference requires careful attention to memory, compute, storage, benchmarking, and optimization. Overall, the chapter concludes that modern SLM deployment frameworks provide accessible APIs and strong baseline performance, but successful serving still depends on measuring, optimizing, and validating models in their real target environment.
The online available documentation for the custom API of the sample FastAPI web application described in this section
The online available ReDoc alternative documentation for the custom API of the sample FastAPI web application described in this section
The end-to-end general workflow for MLC LLM, from model conversion to deployment on all the supported platforms’ native runtimes
The list of supported operating systems, CPUs and GPUs in MLC LLM
Summary
- vLLM uses PagedAttention to manage memory efficiently by storing key-value cache blocks non-contiguously to reduce fragmentation.
- vLLM provides both offline and online serving capabilities with OpenAI-compatible API endpoints for language model deployment.
- vLLM supports quantized models including GGUF format with automatic download and inference capabilities.
- vLLM provides OpenAI-compatible REST API for model serving.
- vLLM provides benchmark scripts to measure serving performance, including throughput, latency, and time-to-first-token metrics.
- FastAPI allows the implementation of high-performance web APIs using Python type hints with automatic documentation generation.
- FastAPI routes define URL endpoints that handle HTTP requests through decorated Python functions.
- Pydantic BaseModel classes validate input and output data structures for FastAPI applications.
- FastAPI applications can integrate Hugging Face models and pipelines for machine learning inference endpoints.
- MLC LLM compiles language models for deployment across multiple platforms using a unified engine and compiler.
- mlc_llm convert_weight transforms model weights into MLC format with specified quantization settings.
- mlc_llm gen_config generates chat configuration files for converted models with appropriate templates.
- mlc_llm compile creates platform-specific model libraries for target devices and operating systems.
- MLCEngine class provides OpenAI-compatible chat completion API for MLC LLM converted models.
- MLC LLM allows model serving through REST API with support for concurrent requests and multiple model hosting.
- Android deployment requires enabling Developer Options and can use MLC LLM framework or alternative approaches.
- mllm framework provides lightweight C++ inference engine for mobile devices with built-in quantization support.
- Termux app enables Python environment setup on Android devices for running Transformers models directly.
FAQ
What deployment and serving environments does Chapter 11 focus on for Small Language Models (SLMs)?
The chapter focuses on practical deployment and inference options for small, customized language models, including vLLM for high-throughput serving, FastAPI for custom REST APIs, MLC LLM for cross-platform and edge deployment, and several approaches for running SLMs on Android devices.What is vLLM and why is it useful for serving SLMs?
vLLM is an open source library for LLM serving and inference. It is useful because it provides high-throughput inference, efficient memory management, continuous batching, streaming outputs, quantization support, distributed inference, OpenAI API compatibility, Hugging Face model integration, and Multi-LoRA support.What problem does PagedAttention solve in vLLM?
PagedAttention improves memory management for LLM serving by reducing fragmentation and redundancy in the key-value, or KV, cache. Unlike older systems that require contiguous memory allocation, PagedAttention allows non-contiguous KV cache storage, which improves memory flexibility, increases effective batch size, and boosts serving throughput.How can vLLM be used for offline inference?
vLLM can be installed as a Python library, usually withpip install vllm. A model can then be loaded from the Hugging Face Hub using the LLM class. For example, loading GPT-2 with LLM("openai-community/gpt2") downloads the configuration, tokenizer, checkpoints, loads the model onto the target device, initializes the KV cache, profiles memory, and warms up the model. Text generation is then performed with generate(), while chat models can use chat().How does vLLM support chat and quantized models?
For chat use cases, vLLM provides thechat() method, which accepts conversations in role-based message format, such as system, user, and assistant. It can run single or batch chat inference and can optionally use custom chat templates. vLLM also supports quantized models, including GGUF models, such as a 4-bit quantized Phi-3 mini model, by loading the quantized model file and its tokenizer and then calling chat() as usual.How is vLLM used for online serving?
vLLM includes an HTTP server compatible with the OpenAI API. A server can be started from the command line with a command such asvllm serve microsoft/Phi-3-mini-4k-instruct --dtype float16 --api-key token-abc123. Once running, clients can call it through the OpenAI Python client by setting the base URL to the vLLM server endpoint, for example http://localhost:8000/v1.Why is benchmarking important after deploying a model with vLLM?
Even though vLLM is highly optimized, real serving performance should be measured after deployment and before client applications use the model. The vLLM repository includes benchmark scripts for measuring latency and throughput. Reports can include successful requests, benchmark duration, input and generated token counts, request throughput, token throughput, time to first token, time per output token, and inter-token latency.How can FastAPI be used to serve an SLM?
FastAPI can expose an SLM through custom HTTP routes. A typical setup creates aFastAPI app, defines Pydantic models for request and response data, initializes a model pipeline or Transformers model, and implements a route such as /generate that accepts a prompt and returns generated text. FastAPI is well suited for ML APIs because it is performant, supports asynchronous operations, validates data with Pydantic, and automatically generates documentation at /docs and /redoc.How does the chapter recommend selecting the best model version for FastAPI deployment?
The chapter recommends benchmarking multiple versions of the same model before deploying it. In the GPT-2 example, the compared versions are the vanilla PyTorch model, an ONNX-converted model, an optimized ONNX model, and an optimized FP16 ONNX model. The benchmark measures average latency across prompts of different token lengths. In the example, the optimized ONNX model performs best up to 512 tokens and is selected as the best candidate for FastAPI deployment.What is MLC LLM and how does it support edge deployment?
MLC LLM is an open source ML compiler and high-performance deployment engine for LLMs. It uses theMLCEngine, which supports deployment across multiple platforms, including REST servers, Python, JavaScript, iOS, and Android. The general workflow involves downloading a Hugging Face model, converting its weights to the MLC format, generating chat configuration, compiling the model library for the target device, and then running inference through the Python API, REST API, or native platform runtime.
Domain-Specific Small Language Models ebook for free
