Overview

1 Why companies self-host large language models

This chapter introduces self-hosting as the practice of running a language model on infrastructure your organization controls instead of sending prompts to an external API provider. It frames the shift as an infrastructure problem rather than a deep machine-learning problem: modern models are downloadable, serving software is open source, and platform teams can now build reliable internal AI services without needing a specialized research staff. The chapter sets expectations for the rest of the book by showing that the goal is not just to run a model, but to operate it as a dependable production service with predictable cost, clear data handling, and manageable operations.

The main argument for self-hosting is built around five forces: privacy and confidentiality, cost at scale, control over latency and availability, vendor concentration risk, and regulation. Each force affects a different part of the serving stack, from where prompts travel and how logs are handled to whether a provider’s pricing, policies, or outages can disrupt a critical business function. The chapter also explains why self-hosting has become realistic now: open-weight models have reached useful quality, open-source inference engines like vLLM make serving practical, and the ecosystem has matured enough that organizations can choose among several serious model families rather than treating external APIs as the only viable option.

Beyond motivation, the chapter maps the landscape and the system itself. It clarifies that self-hosting means controlling the data path, model version, availability, and audit trail, whether the hardware is on-premises, in cloud rented GPUs, or in specialized GPU infrastructure. It distinguishes open-weight from open-source and stresses that licenses and upstream lineage matter just as much as technical performance. Finally, it sketches the layered anatomy of an inference platform, from GPU hardware and model files through the inference engine, API, orchestration, gateway, and observability, while also warning that self-hosting is not always the right choice when traffic is light, the project is still exploratory, the strongest commercial model is required, no team can operate the platform, or none of the five forces apply strongly enough to justify the commitment.

Kubernetes runs programs across several connected machines. Its control plane decides which machine runs each copy of vLLM, restarts copies that fail, and adds machines and copies as traffic grows.
Kubernetes runs programs across several connected machines. Its control plane decides which machine runs each copy of vLLM, restarts copies that fail, and adds machines and copies as traffic grows.
Two data-flow diagrams side by side. Left: application, then public internet, then provider API on provider infrastructure, with data leaving the systems the company controls. Right: application, then company gateway, then inference engine on company-controlled GPUs, with data staying inside the systems the company controls.
Two data-flow diagrams side by side. Left: application, then public internet, then provider API on provider infrastructure, with data leaving the systems the company controls. Right: application, then company gateway, then inference engine on company-controlled GPUs, with data staying inside the systems the company controls.
In a dense model, every token passes through the entire feed-forward layer of each transformer block. In an MoE model, attention remains dense, but in MoE transformer blocks the feed-forward layer is replaced by a set of experts. A router selects a small subset of them for each token (here, E1 and E5), and their outputs are combined. This allows the model to scale to many more parameters without activating all of them for every token. The experts typically need to reside in accelerator memory, possibly distributed across multiple GPUs.
In a dense model, every token passes through the entire feed-forward layer of each transformer block. In an MoE model, attention remains dense, but in MoE transformer blocks the feed-forward layer is replaced by a set of experts. A router selects a small subset of them for each token (here, E1 and E5), and their outputs are combined. This allows the model to scale to many more parameters without activating all of them for every token. The experts typically need to reside in accelerator memory, possibly distributed across multiple GPUs.
The inference platform, shown as layers from bottom to top: (1) GPU hardware and memory; (2) the model files; (3) the inference engine that runs the model; (4) the API that receives requests; (5) software that places, restarts, and scales programs across computers; (6) a gateway that routes requests and enforces limits; and (7) monitoring, cost tracking, and governance across all layers. The solid arrow shows requests flowing down from the gateway toward the hardware; the dashed arrow shows answers returning. The orchestration layer and the observability plane manage and observe that path rather than carry it.
The inference platform, shown as layers from bottom to top: (1) GPU hardware and memory; (2) the model files; (3) the inference engine that runs the model; (4) the API that receives requests; (5) software that places, restarts, and scales programs across computers; (6) a gateway that routes requests and enforces limits; and (7) monitoring, cost tracking, and governance across all layers. The solid arrow shows requests flowing down from the gateway toward the hardware; the dashed arrow shows answers returning. The orchestration layer and the observability plane manage and observe that path rather than carry it.
A single request traveling through the platform. (1) The user's application sends the question to the serving API, the platform's single visible address. (2) The API server prepares the request and converts the text into tokens, the numeric units the model reads. (3) The inference engine schedules the request onto the GPU alongside requests from other users. (4) The GPU runs the model: it first processes the whole prompt, then produces the answer one token at a time. (5) Each token streams back through the same path, so the user sees the answer appear word by word while it is still being generated. The gateway, orchestration, and monitoring layers surround this path in production, routing the request to the right server and recording what happened.
A single request traveling through the platform. (1) The user's application sends the question to the serving API, the platform's single visible address. (2) The API server prepares the request and converts the text into tokens, the numeric units the model reads. (3) The inference engine schedules the request onto the GPU alongside requests from other users. (4) The GPU runs the model: it first processes the whole prompt, then produces the answer one token at a time. (5) Each token streams back through the same path, so the user sees the answer appear word by word while it is still being generated. The gateway, orchestration, and monitoring layers surround this path in production, routing the request to the right server and recording what happened.

Summary

  • Five forces push model serving in-house: data privacy, cost at scale, control, vendor concentration risk, and regulation. Identify the ones that apply to you; they shape what you build.
  • Self-hosting means running models on infrastructure your organization controls: your own data center, rented GPU servers, or a specialized GPU provider. The defining property is control.
  • The open-weight landscape is best read as family lineups, not as individual models. The dense-versus-Mixture-of-Experts distinction matters most: MoE models are memory-hungry but computation-frugal, which changes sizing and the way work is split across GPUs.
  • Open-weight is not the same as open-source. A license can disqualify a model before any quality test is run, so it should be evaluated first.
  • An inference platform is ordinary layered platform engineering applied to an unusual workload: hardware, model artifacts, engine, API, orchestration, gateway, and a cross-cutting observability and governance plane.
  • Self-hosting is the wrong choice at low volume, during early prototyping, when the strongest commercial models are a hard requirement, or when no team will own the platform.

FAQ

Why are companies choosing to self-host large language models instead of using a hosted API?

Companies self-host to keep sensitive data under their own control, reduce costs at higher usage, improve control over latency and availability, avoid dependence on a single vendor, and meet regulatory requirements more easily.

What are the five main forces that drive self-hosting decisions?

The five forces are data privacy and confidentiality, cost at scale, control, vendor concentration risk, and regulation. Most organizations self-host because of some combination of these factors rather than just one.

What does self-hosting an LLM actually mean?

Self-hosting means downloading a model and running it on infrastructure your organization controls, such as your own data center, rented GPUs, or dedicated cloud capacity. The key idea is control over the systems that handle the data and serve the model.

How is self-hosting different from using a managed model service like Bedrock or Vertex AI?

Managed services may keep requests inside your cloud account, but the provider still runs the model and charges per token. With self-hosting, you control the model runtime, infrastructure, access, and data path directly.

Why is data privacy such a strong reason to self-host?

When prompts are sent to a third-party API, sensitive text leaves your environment. For highly confidential workloads like legal, healthcare, or financial data, keeping prompts and outputs inside infrastructure you control is often essential.

How can self-hosting reduce costs?

API pricing grows with every token processed, so costs can rise sharply as usage scales. Self-hosting replaces per-token charges with infrastructure costs, which can become cheaper when the system has steady or high volume and available GPU capacity.

Why does self-hosting improve control?

Self-hosting lets you control latency, uptime, model versioning, and customization. It also avoids surprises from provider outages, model changes, or policy restrictions that can affect hosted APIs.

What open-weight model families are most relevant for production self-hosting?

The chapter highlights major families such as Llama, Qwen, Mistral, DeepSeek, Kimi, Gemma, gpt-oss, GLM, and Nemotron. These are the families most commonly considered for real production deployments.

What is the difference between dense and Mixture-of-Experts (MoE) models?

Dense models use all of their parameters for every token. MoE models route each token to only a few specialized experts, so they may need a lot of memory but often use less computation per token.

When is self-hosting the wrong choice?

Self-hosting is usually not the right choice when traffic is low or irregular, you are still prototyping, you need the very strongest commercial models, no team will own operations, or none of the five forces applies. In those cases, a hosted API is often the better option.

pro $24.99 per month

  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose one free eBook per month to keep
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime

lite $19.99 per month

  • access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more


Close
choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Self-Hosting LLMs at Scale Self-Hosting LLMs at Scale ebook for free
Close
choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Self-Hosting LLMs at Scale Self-Hosting LLMs at Scale ebook for free