14 Advanced Components for LLM Applications
This chapter extends earlier work on retrieval-augmented generation and agentic AI by showing how small language models can support more robust, domain-specific applications. It focuses on three advanced capabilities: enriching RAG with knowledge graphs, combining retrieval systems with autonomous agents, and adding short- and long-term memory to conversational or agent-based systems. Across these topics, the chapter emphasizes practical, local-first implementations using open source tools and SLMs rather than relying exclusively on large commercial models.
A major theme is graph-based retrieval, where a knowledge graph captures entities, relationships, and communities extracted from source documents. This approach helps overcome limitations of standard vector-based RAG, especially for questions requiring multi-hop reasoning or connections across separate pieces of information. The chapter walks through a naïve implementation using a local Mistral model, structured outputs, NetworkX, community detection, community summaries, and a global search strategy that generates intermediate answers from graph communities before combining them into a final response. It also discusses Microsoft’s open source GraphRAG implementation, its global and local search modes, persistence options, storage limitations, and the need for RAG-specific evaluation metrics such as context precision, context recall, faithfulness, and answer relevancy.
The chapter then shows how RAG can be combined with agentic AI so an agent can choose among retrieval tools, perform keyword search, semantic search, hybrid search, or web search, and break complex tasks into multiple steps. Using SmolAgents, LangChain, LanceDB, BM25, Hugging Face embeddings, and a small instruction model, the example demonstrates how an agent can select the right tool depending on the user query and combine local knowledge with external information. Finally, the chapter introduces memory management for LLM applications through Memoripy, explaining how past interactions can be stored, embedded, organized into concepts and graph associations, retrieved by relevance, and reinforced or decayed over time. This enables chatbots, RAG systems, and agents to generate more context-aware responses while potentially reducing repeated retrieval and model calls.
The logical blocks and workflow for the GraphRAG proposal from Microsoft.
The action movie graph for the example data in this section
The logical schema of a basic Agentic RAG system
Output of the CodeAgent for search at the first execution step
The output of the updated CodeAgent for search after the first execution step
The CodeAgent’s output that shows the proper tool choice (hybrid search on the local LanceDB database) and correct Python code generation for the given query
The CodeAgent’s output that shows the proper tool choice (web search) and correct Python code generation for the given query
The CodeAgent’s output on a query that requires using multiple tools
Summary
- Graph RAG enhances traditional RAG by using knowledge graphs to capture entity relationships and enable multi-hop reasoning across document collections.
- Knowledge graphs organize information as nodes representing entities and edges representing relationships between those entities.
- Microsoft's GraphRAG approach builds entity knowledge graphs from source documents and pre-generates community summaries for related entity groups.
- Ollama structured outputs constrain model responses to specific JSON schemas using Pydantic classes for consistent data formatting.
- NetworkX library provides graph creation and manipulation capabilities for building knowledge graphs from extracted entities and relationships.
- The Leiden algorithm identifies graph communities by optimizing modularity through iterative node movement and community aggregation.
- Community detection groups densely connected entities to create meaningful clusters for downstream summarization tasks.
- Community summaries provide high-level overviews of entity groups that improve global search quality in GraphRAG systems.
- Global search generates intermediate answers for each community summary then combines them into final responses.
- Local search combines knowledge graph entities with traditional document embeddings for hybrid retrieval approaches.
- Microsoft's open source GraphRAG implementation supports only OpenAI models but provides complete indexing and querying workflows.
- RAG evaluation requires specialized metrics like context precision, context recall, faithfulness, and answer relevancy rather than traditional NLP metrics.
- Agentic RAG systems combine autonomous agents with retrieval capabilities to perform multi-step information gathering and reasoning.
- SmolAgents CodeAgent can autonomously select appropriate tools for different search strategies based on query requirements.
- BM25 algorithm ranks documents using term frequency, inverse document frequency, and document length for keyword-based retrieval.
- Hybrid search combines keyword-based BM25 retrieval with semantic similarity search to improve retrieval quality.
- LangChain EnsembleRetriever assembles results from multiple retrievers with configurable weights for different search methods.
- Custom retriever classes extend LangChain BaseRetriever to implement domain-specific search functionality.
- Memory management systems track short-term conversation history and long-term user interactions across sessions.
- Memoripy provides graph-based memory storage with concept extraction, embedding generation, and spreading activation for relevance retrieval.
- Decay and reinforcement mechanisms promote frequently accessed memories while degrading unused interactions over time.
FAQ
What is Graph RAG, and why is it useful compared with baseline RAG?
Graph RAG augments Retrieval Augmented Generation with a knowledge graph. A baseline RAG system usually retrieves text chunks based on semantic similarity, which works well for many questions but can struggle with complex queries requiring multi-hop reasoning or connections across dispersed facts. Graph RAG represents knowledge as entities, attributes, and relationships, allowing retrieval and generation to use graph structure, community summaries, and relationships between concepts. This can improve answer comprehensiveness, diversity, and reasoning over private or domain-specific corpora.
How does Microsoft’s GraphRAG approach build and use a graph index?
Microsoft’s GraphRAG approach builds a graph index in two main indexing stages. First, an LLM extracts entities and relationships from source documents to create an entity knowledge graph. Second, it detects groups of closely related entities, called communities, and pre-generates summaries for those communities. At query time, community summaries are used to generate partial responses, and those partial responses are combined into a final answer for the user.
What are graph communities in Graph RAG?
Graph communities are groups of entities or nodes that are more densely connected to each other than to the rest of the graph. For example, in a movie knowledge graph, an “Action Movies” community might include action films, actors, directors, and the “Action” genre, connected by relationships such as “features actor” or “directed by.” Communities help downstream tasks such as trend analysis, recommendations, and query-focused summarization.
Why are structured outputs useful when extracting entities and relationships with a local SLM?
Structured outputs help ensure that the model returns data in a predictable format. In the chapter’s example, Mistral 7B sometimes failed to follow the requested plain-text format for entities and relationships, even with temperature set to zero. By using Ollama’s structured output support with a Pydantic schema, the response can be constrained to JSON containing two lists: entities and relationships. This makes parsing easier and reduces formatting-related errors before building the knowledge graph.
What role does NetworkX play in the custom Graph RAG implementation?
NetworkX is used to build and manipulate the knowledge graph. Extracted entities become graph nodes, and extracted relationships become graph edges. The implementation builds partial graphs for individual chunks and then combines them into a larger graph using NetworkX functions such as nx.Graph(), add_nodes_from, add_edge, and nx.compose. NetworkX can also support graph analysis and community detection workflows.
What is the difference between Global Search and Local Search in Microsoft’s GraphRAG?
Global Search answers user queries using the community summaries generated during indexing. It is the central strategy described in Microsoft’s GraphRAG paper and is the approach implemented in the chapter’s naïve local example. Local Search combines relevant data from the AI-extracted knowledge graph with raw document text chunks and embeddings, requiring a vector database in addition to graph elements. Microsoft’s implementation supports both, but the chapter focuses mainly on Global Search.
Which metrics are recommended for evaluating RAG and Graph RAG systems?
The chapter recommends RAG-specific metrics instead of relying only on traditional NLP metrics such as BLEU and ROUGE. For retrieval, useful metrics include Context Precision, which measures how much retrieved content is relevant, and Context Recall, which measures how much relevant available content was retrieved. For generation, useful metrics include Faithfulness, which measures whether generated facts are supported by retrieved documents, and Answer Relevancy, which measures how relevant the answer is to the original query.
What is Agentic RAG?
Agentic RAG combines Retrieval Augmented Generation with AI agents. In a traditional RAG system, retrieval and generation are usually fixed steps. In Agentic RAG, an agent can autonomously plan multi-step retrieval, choose among tools, adapt queries based on retrieved information, and combine multiple information sources such as vector databases, keyword search, and web search. This is useful for customer support, research assistance, smart knowledge management, and domain-specific problem-solving.
How does the chapter implement Agentic RAG using SmolAgents?
The chapter uses SmolAgents with custom tools for different retrieval strategies. A BM25SearchTool performs keyword-based search over local document chunks. A SemanticSearchTool performs vector similarity search over LanceDB using embeddings from all-MiniLM-L6-v2. A HybridSearchTool combines BM25 and semantic retrieval using LangChain’s EnsembleRetriever. The agent can also use a web search tool such as DuckDuckGoSearchTool, allowing it to choose local search, web search, or multiple tools depending on the user query.
How does Memoripy support long- and short-term memory for LLM applications?
Memoripy manages past user interactions for chatbots, RAG systems, and agents. It stores interactions with prompts, outputs, embeddings, extracted concepts, timestamps, access counts, and decay factors. It supports short-term and long-term memory, contextual retrieval based on embeddings and concepts, graph-based associations, hierarchical clustering, decay of older memories, and reinforcement of frequently accessed memories. In the chapter’s example, Memoripy is used with local Ollama models: Phi-3 mini for generation and mxbai-embed-large for embeddings, with interaction history stored in a JSON file.
Domain-Specific Small Language Models ebook for free
