12 Designing LLM-powered systems

This chapter explains how large language models extend, rather than replace, established MLOps foundations. Core platform capabilities—containerization, Kubernetes orchestration, CI/CD, and system monitoring—remain essential, but LLM applications introduce new operational realities: non-deterministic outputs that break exact-match testing, token and context-window budgeting, multi-step reasoning chains that require traceability, prompt engineering as a versioned, testable artifact, and safety considerations that go beyond accuracy. The economic model also shifts to token-based usage, pushing teams to manage latency, quality, and cost together. The message is clear: LLMOps is an evolution of proven ML engineering principles adapted to the unique variability, scale, and risk profile of generative systems.

To make these ideas concrete, the chapter builds DakkaBot, an enterprise RAG assistant for company documentation. It covers the full stack: ingesting and chunking documents, creating asymmetric embeddings for documents vs. queries, indexing with a vector database (e.g., FAISS), and retrieving semantically relevant context. The augmentation step assembles and formats that context for the LLM, while generation is guided by carefully constructed prompts and tuned sampling parameters (often with low temperature for consistency). LangChain provides interchangeable, production-ready components and orchestration; LangGraph is highlighted for more complex, branching workflows. The system is wrapped in a chat UI with Chainlit, delivering streaming responses, source attribution, and robust error handling. Throughout, the chapter emphasizes composability, token-budget management, prompt/version governance, and seamless fit with existing infrastructure, from autoscaling to artifact tracking.

Production readiness centers on observability, safety, and cost control. The chapter introduces LLM-specific tracing with LangFuse to capture retrieval quality, prompt inputs, token usage, latency breakdowns, and conversation-level analytics—enabling semantic evaluation over brittle string comparisons. It prescribes versioning for prompts and retrieval parameters, A/B testing of variants, and governance over model/provider choices. Security guardrails address prompt injection, jailbreaks, and leakage risks, complemented by adversarial testing tools and input validation. Operational hardening includes rate limiting, retries, semantic caching, fallbacks across providers/models, and index versioning with rollback. By combining these patterns—composable RAG architecture, streaming UX, prompt and artifact management, and deep observability—the chapter provides a pragmatic blueprint for designing reliable, cost-aware, and trustworthy LLM-powered systems.

The mental map where we are focusing on building our RAG application (12).

Traditional ML models and LLM applications coexist in the same cluster, managed by identical infrastructure components. The same Horizontal Pod Autoscaler that scales your XGBoost recommendation engine also scales your LangChain RAG application. No separate infrastructure needed—just different workload types sharing proven Kubernetes orchestration.

LangChain chains link components where each chunk's output becomes the next chunk's input.

LangChain (left) provides sequential chains ideal for straightforward RAG pipelines, while LangGraph (right) enables complex graph-based workflows with conditional branching, parallel execution, and iterative refinement loops.

Build-time setup (top) ingests documents into a vector database and configures the LLM. Query-time flow (middle) handles user questions through embedding, retrieval, augmentation, and generation. Implementation requires three lines of LangChain code.

Google AI Studio API key generation interface with quickstart guide for Gemini model integration.

Successful launch with showing the interface that requires no frontend development.

LangFuse dashboard showing DakkaBot's traces, costs, and latency metrics.

LangFuse displays detailed traces including costs, inputs, outputs and intermediate results.

• LLMs don't replace traditional MLOps infrastructure—they extend it. Your Kubernetes clusters, monitoring pipelines, and CI/CD systems remain essential, but they must support new components like vector databases, prompt management, and specialized observability tools.
• The shift from single model deployments to composite systems fundamentally changes how you approach testing and debugging. Your "model" becomes an orchestrated pipeline of LLMs, embeddings, retrievers, and prompts, each with independent failure modes that require holistic monitoring.
• Token-based pricing models require new optimization strategies. Unlike traditional compute costs that scale with infrastructure, LLM costs scale with usage patterns. Understanding prompt efficiency and context optimization becomes as important as traditional performance tuning.
• The composable nature of LLM applications enables both flexibility and complexity. While you can swap embedding models or vector databases without rewriting integration code, the increased surface area for failures demands more sophisticated observability and error handling strategies.

FAQ