13 Creating End-to-end LLM applications
This text explains that standalone small or large language models are often not enough to deliver real business value on their own. Instead of treating the model as the whole solution, it argues that the model should be designed as one component in a broader end-to-end system that includes user interfaces, data sources, context enrichment, post-processing, security, scalability, reliability, and compliance. The central message is that domain-specific small language models can be more practical than ever-larger general models because they are easier to specialize, cheaper to run, simpler to control, and better suited to regulated or privacy-sensitive environments.
The text then walks through how to build a retrieval-augmented generation system using only open-source small models and local components. It describes the full workflow: extracting text from documents, cleaning and splitting it into sentences and chunks, generating embeddings, storing or indexing those embeddings, retrieving semantically relevant chunks for a user query, and passing the retrieved context to a generator model to produce an answer. It first demonstrates retrieval without a vector database to show the underlying mechanics of similarity search, then introduces a vector database to make retrieval more scalable, reliable, and performant. The examples emphasize that good chunking, consistent embedding models, appropriate similarity search, and careful prompt formatting are essential to answer quality.
The final part introduces agentic AI as a more advanced application pattern where a language model can reason about a task, select tools, act through those tools, and interact with users or other systems. It distinguishes real agents from simple automation that merely wraps a model call, then demonstrates how a local small model can be used with an agent framework to perform tasks such as searching for flights, selecting seats, and booking a ticket through custom tools. The text highlights best practices for building reliable agents: use instruction-tuned models, keep workflows simple, validate tool inputs, make deterministic components handle deterministic work, limit model responsibilities to tasks it can perform reliably, and evaluate whether the chosen small model is strong enough for the agentic workload.
The sample business goal for our imaginary Formula 1 racing team: build a competitive car, where the engine is an important but not the only one component[2]
The wrong pattern that would certainly lead to failure as a Formula 1 racing team: building a mammoth engine and putting it on wheels, ignoring integration with the other components and the main business scope
The generic workflow of a RAG system
The generic workflow of a RAG system (same as in figure 13.3, but presented here again to make easier to follow the rest of this section.
The logical workflow of CodeAgent
Comparison between using Text, JSON and a code interpreter as action on the same example task. On the top, a comparison of the generated steps. On the bottom, quantitative experiment results for different LLMs.
Summary
- Language models alone generate limited business value compared to integration within complex systems that include external knowledge and tools.
- RAG systems enhance language models by combining retrieval from external knowledge bases with text generation capabilities.
- Document chunking splits large texts into smaller segments to prevent overloading embedding and generator model capacity limits.
- PyMuPDF extracts content from PDF documents while spaCy provides sentence boundary detection for text preprocessing.
- Sentence Transformers library converts text chunks into dense vector representations using encoder-only transformer architectures.
- Cosine similarity measures semantic relationships between query embeddings and document embeddings for relevant content retrieval.
- Vector databases like LanceDB store embeddings with approximate nearest neighbor algorithms for fast similarity search.
- LanceDB provides serverless vector storage with automatic versioning and zero-copy capabilities for embedding management.
- Embedding models must be identical for both knowledge base creation and query processing to ensure consistent vector spaces.
- Generator models receive enriched prompts containing user queries plus retrieved context to produce informed responses.
- Prompt formatting functions combine user queries with retrieved context items using model-specific chat templates.
- AI agents autonomously perform tasks and make decisions by reasoning about problems and executing actions through external tools.
- Smolagents framework enables local agent development using small language models with Python-based tool execution.
- CodeAgent writes and executes Python code snippets for each action step rather than generating JSON-formatted instructions.
- Tool functions require docstrings and type hints to enable automatic conversion into agent-usable capabilities.
- ReAct agents integrate reasoning and action by analyzing problems and iteratively working toward solutions using available tools.
- Custom tools extend agent capabilities beyond default toolsets for domain-specific task automation.
- Agent planning involves breaking complex tasks into sequential steps and selecting appropriate tools for execution.
- Multi-step workflows allow agents to chain tool outputs together for comprehensive task completion.
FAQ
Why does the chapter argue that standalone LLMs or SLMs are often not enough to solve business problems?
A standalone language model usually receives a prompt, generates text, and returns a result, but real business value often comes from a complete system around the model. The chapter compares this to Formula 1: the engine is important, but the car succeeds only when the engine, tires, electronics, mechanics, rules, and design goals work together. Similarly, an LLM/SLM application usually needs user interfaces, data sources, context enrichment, post-processing, security, scalability, reliability, and compliance. Small domain-specific models can be easier to specialize, control, secure, and integrate into this larger system.What is Retrieval Augmented Generation, or RAG, and why is it useful with small language models?
RAG is a system pattern that combines a language model with an external knowledge base. Instead of relying only on what the model learned during training, the system retrieves relevant document chunks and adds them to the prompt before generation. This improves answer relevance, reduces dependence on model memory, and helps use up-to-date or domain-specific information. The chapter emphasizes that RAG can be built entirely with SLMs: one small model can create embeddings for retrieval, while another small model generates the final answer.What are the two main phases of a RAG system?
A RAG system has an offline phase and an online phase. In the offline phase, documents are parsed, cleaned, split into chunks, transformed into embeddings, and stored in a vector database or another persistent format. This happens when the knowledge base is created or updated. In the online phase, a user query is embedded using the same embedding model, similar document chunks are retrieved, and those chunks are added as context for the generator model to produce an answer.Why is document chunking important in a RAG pipeline?
Document chunking breaks large documents or pages into smaller text segments that are easier for embedding and generation models to handle. It helps avoid exceeding the token limits of the embedding model and the context window of the generator model. In the chapter’s example, PDF pages are extracted with PyMuPDF, split into sentences with spaCy, grouped into chunks of about 10 sentences, and then converted into paragraph-like chunks with metadata such as page number, character count, word count, and approximate token count.Which embedding model is used in the chapter’s RAG example, and what does it do?
The chapter usessentence-transformers/all-mpnet-base-v2, loaded through the Sentence Transformers library. This model maps sentences and paragraphs into a 768-dimensional vector space. Texts with similar meanings are placed closer together in that space, which enables semantic search. The same embedding model must be used both for embedding the document chunks and for embedding user queries.What is the difference between keyword search and similarity search in RAG?
Keyword search looks for exact or near-exact text matches between a query and documents. Similarity search compares numerical embeddings, so it can retrieve text that is semantically related to the query even if it does not contain the exact same words. The chapter discusses dot product and cosine similarity as common vector similarity algorithms. Cosine similarity is typically used for text because it compares vector direction, which captures semantic closeness.Why does the chapter introduce a vector database, and which one is used in the example?
A vector database stores embeddings and associated metadata, builds indexes over them, and enables fast, scalable similarity search. Persisting embeddings to a CSV file can work for small local examples, but larger knowledge bases require a proper vector store. The chapter uses LanceDB because it is easy to set up locally, supports persistent storage, vector similarity search, full-text search, SQL-like access, Python and JavaScript/TypeScript support, automatic versioning, and integration with tools such as LangChain, LlamaIndex, Apache Arrow, Pandas, and Polars.Which generator model is used for local RAG generation in the chapter?
The chapter usesmicrosoft/Phi-3-mini-4k-instruct-gguf, a 4-bit quantized GGUF version of Phi-3 Mini 4K Instruct. It is loaded with llama-cpp-python so the end-to-end example can run locally in Python. The retrieved chunks are inserted into a custom prompt, formatted with the model’s chat template, and then passed to the generator to answer the user query.What makes an AI system an agent rather than a simple LLM-powered workflow?
An AI agent can reason, act, and interact. It analyzes a task, decides what steps are needed, selects suitable tools, executes actions through tools or APIs, and may interact with users or other agents. The chapter contrasts a normal travel-planning chatbot, which merely returns accommodation and flight suggestions, with an agentic system that searches multiple sources, combines flights and accommodation, suggests related services, listens to feedback, and can proceed with bookings.What is Smolagents, and how is it used to build local agents with SLMs?
Smolagents is a small open source Hugging Face framework for building LLM-based agents. It supports models from Hugging Face, OpenAI-compatible endpoints, Azure OpenAI, Together AI, LiteLLM, Ollama, and local Transformers models. The chapter focuses onCodeAgent, which reasons about tasks and writes executable Python code to call authorized tools. In the flight-booking example, a local TransformersModel using HuggingFaceTB/SmolLM2-1.7B-Instruct is combined with tools such as search_for_flights, find_seat, and book_flight_ticket. The agent plans the steps, calls the right tools, and returns the final result.
Domain-Specific Small Language Models ebook for free
