5 Building knowledge bases with RAG

This chapter explains why AI agents need access to external knowledge bases instead of relying only on the model’s internal knowledge. Real tasks often involve private documents, code repositories, or large collections of files that must be searched selectively and added to context. The chapter frames retrieval-augmented generation as the overall process of finding relevant information, passing it to the LLM, and using that retrieved evidence to produce better answers.

It then compares several search approaches and shows when each one is useful. Keyword search is best for exact matches such as function names or error codes, while vector search is stronger for natural language and semantic similarity because embeddings place related meanings close together in vector space. Graph search is introduced for relationship-heavy questions, and structure-based search is presented as a practical method for exploring organized data like codebases and zip archives by using file names, folders, and other structural clues.

The chapter also walks through a full hands-on implementation of vector search, including embedding text, splitting long documents into chunks, calculating cosine similarity, and understanding why vector databases are needed for scale. A practical exercise shows how chunking and vector search can reduce thousands of tokens of web search content down to a tiny relevant subset. Finally, the chapter extends the agent with the Callback pattern, adding approval before dangerous tool use and automatic compression of search results, demonstrating that agents can gain powerful new behaviors without changing their core logic.

Book structure overview - Chapter 5 in focus.

The overall flow of RAG (Retrieval-Augmented Generation). When a user question comes in, the system searches for relevant information from the Knowledge Base, then passes the search results along with the question to the LLM to generate a response.

Types of Search - External/Internal repositories, Keyword/Vector/Graph search, and Structure-based search.

Keyword Search - While "sofa" and "couch" are similar words, a keyword search cannot determine their similarity without separate processing. Common expressions like "is" that appear frequently everywhere are not considered important (shown with triangles). The left example is judged relevant due to overlapping "sofa," but the right example is judged irrelevant despite semantic relevance.

Vector Search Method - Determines relevance by converting data into vectors. Using a well-trained embedding model, it can determine that "sofa" and "couch" are similar without synonym processing.

Graph Search Method - Extracts entities and relationships between entities from sentences and represents them as a graph. Each entity and relationship is sometimes converted to embedding vectors to utilize both vector search and graph search together.

Points where Callbacks are inserted in the agent execution flow. LLM callbacks are called before and after think(), and tool callbacks are called before and after each tool execution within act().

Tool execution approval flow in the Human in the Loop pattern. When a user requests "Clean up the project folder," the agent pauses when it tries to call the delete_file tool and requests approval from the user.

• Internal data requires different approaches than web search. Single files can be read directly into context, but multiple files need exploration, and large-scale data requires search mechanisms to extract only relevant portions.
• Four search methods serve different needs: keyword search for exact matching, vector search for semantic similarity, graph search for relationship traversal, and structure-based search for navigating organized file systems. Production systems often combine multiple methods.
• Vector search converts text to numerical vectors where semantic similarity becomes geometric distance. The pipeline involves chunking long documents, generating embeddings, and calculating cosine similarity to find the most relevant content.
• Structure-based search leverages file system organization as information itself. Folder names, file names, and directory hierarchies provide meaningful metadata that agents can navigate using simple tools like list_files and read_file.
• The Callback pattern extends agent behavior without modifying core logic. before_tool_callback enables Human in the Loop approval for dangerous operations, while after_tool_callback enables automatic compression of search results.
• Context engineering through search dramatically reduces token usage. In our exercise, extracting only relevant chunks reduced over 37,000 tokens to just 304 tokens, achieving a 99% savings rate while preserving the information needed to answer the query.

FAQ