Overview
8 Deploying agents and agentic systems
This chapter explains how agents move from prototypes to production systems by focusing on practical deployment choices. It emphasizes that the way an agent is consumed often determines how it should be deployed, whether it is embedded directly in an application, wrapped as an API service, or exposed as a tool for other agents through protocols such as MCP or A2A. The chapter also shows how real-time browser-based agents work well for responsive, voice-driven interactions, while slower or more isolated workloads are better handled outside the client.
It then develops containerization as the preferred path for scalable, maintainable agent systems. By separating concerns into microservices and packaging agents in Docker containers, developers can upgrade, swap, and scale individual components more easily. The chapter demonstrates building a single-agent container, orchestrating multiple agents with Docker Compose, and even exposing local services for temporary external access with tunneling tools. Throughout, it highlights that the simplest runtime that satisfies latency and tool needs is usually the best starting point.
Finally, the chapter broadens from deployment mechanics to production discipline. It covers key operational concerns such as tracing, metrics, logging, timeouts, fallbacks, caching, idempotency, cost control, and model routing, all of which help keep systems reliable and affordable. It closes with a strong security and governance perspective, urging least-privilege access, secret management, sandboxed tools, prompt-injection defenses, policy enforcement, and human approval for risky actions so agentic systems can be shipped safely and responsibly.
shows three simple patterns for deploying and consuming agents. From embedded agents, a microservice API is accessible or used as a tool through other agents.
Connecting to a real-time model using a RealTime Agent object in a web browser. Allows for vocal interaction with the agent hosted in the browser.
connecting the real-time voice agent to the API image generation agent as a tool and then generating images.
There are several ways afrontend agent may consume containerized microservice agents as tools through an API or as MCP servers.
Docker Desktop interface for managing containers, allowing a user to start/stop containers, delete containers, and images.
shows a set of containers orchestrated through a Docker Compose file.
illustrates how external tunneling options can expose locally running agent services to external users. The Actor represents an external network user accessing an agent service. First the user browses to the tunneling service address and then routed to the a developers local machine.
a helpful decision flowchart for deciding agent deployments.
The practical front-door agent deployment pattern used for user-facing agents and applications
Summary
- Agent consumption drives deployment: embed for ultra‑low latency UX, wrap as a synchronous API for request/response tasks, or run as event‑driven workers for long jobs and retries.
- Realtime agents in the browser (WebRTC/WebSocket) deliver barge‑in speech, token streaming, and the most responsive experiences—keep tools simple or proxy them server‑side.
- Microservices + containers cleanly separate concerns; agents make ideal microservices because they’re self‑contained and easy to scale, swap, and version.
- Dockerizing agent APIs standardizes runtime and dependencies; Compose lets you stand up multi‑agent stacks (UI, worker agents, tool services) with one command.
- External tunneling (e.g., localtunnel) turns local prototypes into shareable demos without full cloud deployment—useful for POCs and quick pilots.
- Choose the “wire” by latency and fit: WebRTC/WebSockets for realtime, HTTP+SSE for streamed request/response, and message buses for decoupled background work.
- Front‑door/orchestrator patterns route user intents to specialized worker agents; keep the front‑door light and push complexity into typed, well‑scoped workers.
- State and idempotency matter: store short‑term chat state separately from long‑term knowledge, and make tool calls idempotent to enable caching, replay, and resilience.
- Release engineering applies to agents: version prompts, tools, and models; promote with gates; pin exact model/tool versions for reproducibility and incident debugging.
- Observability is non‑negotiable: trace from UI → gateway → agent → tools → model; track latency, cost, and success metrics; prefer structured logs with PII redaction.
- Reliability patterns—timeouts, fallbacks, circuit breakers, and graceful degradation—keep systems useful even when tools or models misbehave.
- Cost control comes from routing by intent, trimming context, and caching deterministic results—lower tokens often means lower latency, too.
- Security, safety, and governance must be built‑in: threat‑model surfaces, enforce least privilege, manage secrets correctly, sandbox tools, and defend against prompt‑injection with schema‑first tool contracts and instruction hierarchies.
- With deployment patterns, observability, and safety in place, agents graduate from demos to dependable, production‑ready systems.
AI Agents in Action, Second Edition ebook for free
