This chapter introduces agentic RAG as a retrieval-augmented generation approach where multiple specialized and generic retrieval agents work together to answer user questions. It motivates the need for agentic RAG by contrasting unstructured retrieval (e.g., vector search) with the added value of structured data and by highlighting real-world complexity that often requires narrowly scoped, high-precision retrievers. The system centers on three foundational parts: a Retriever Router that selects the best tool(s), a set of Retriever Agents (from catch-all to highly specialized), and an Answer Critic that validates completeness and correctness before responding, ensuring consistent quality in production.
The chapter outlines how agentic RAG is implemented using LLM tool-use/function-calling to route questions to the right retrievers and supply the correct parameters. Generic agents such as vector similarity search and text2cypher serve broad needs, while domain-specific retrievers address difficult, high-value queries discovered over time. A sequential processing strategy enables continuous query updating: answers from earlier sub-questions are fed back to rewrite later questions into more specific, atomic forms, improving retrieval precision. A tool-call handler executes chosen retrievers, while careful tool descriptions and parameter schemas help the LLM make accurate routing decisions.
Quality control is handled by an Answer Critic, which checks whether the original question has been fully addressed; if not, it proposes targeted follow-up questions and triggers another retrieval round with clear exit criteria to avoid loops. The main flow combines question rewriting, routing, retrieval, and critique, then assembles a final answer under strict grounding rules (use only provided context, no fabrication). The chapter emphasizes incrementally adding specialized retrievers, leveraging structured data (e.g., graphs), and adopting disciplined prompts and interfaces so that agentic RAG remains robust, accurate, and scalable in real-world applications.
The data flow for a application using the Agentic RAG.
Summary
Agentic RAG is a system where a variety of retrieval agents are available to retrieve the data needed to answer the user question.
The main interface to an agentic RAG system is usually some kind of use case or retriever router which job is to find the best suited retriever (or retrievers) to perform the task at hand.
The foundational parts of an agentic RAG system are Retriever Agents, Retriever Router and Answer Critic.
The main parts of an agentic RAG system can be implemented using an LLM with tools/function calling support.
The Retriever Agents can be generic or specialized and should be added over time as needed to improve the performance of the application.
The Answer Critic is a function that takes in the answers from the retrievers and checks if the original question is answered correctly.
What is agentic RAG?An agentic RAG system gives the model the ability to act on a user’s behalf by selecting from multiple retrieval agents to gather the data needed to answer a question. In this chapter, the “agency” is focused on choosing the right retriever(s) and verifying whether the retrieved context fully answers the question.Why do we need agentic RAG?It’s useful when you have diverse or complex data sources and need the best retriever for each query. Agentic RAG helps maintain high performance and consistent answer quality in production by combining specialized retrievers with a routing and validation layer.How does structured data factor into agentic RAG?Structured data carries valuable information in its relationships and schema, so adding structure incrementally can improve retrieval quality. Generic tools like text2cypher work well on graphs, but complex schemas often benefit from specialized retrievers for reliability.What are the core components of an agentic RAG system?- Retriever Router: Chooses the best retriever(s) for a question. - Retriever Agents: The actual retrieval tools (generic or specialized). - Answer Critic: Checks if the original question is fully and correctly answered and can trigger follow-ups.What are Retriever Agents and when should I create specialized ones?Retriever Agents range from broad tools (vector similarity search, text2cypher) to narrow, hard-coded queries (for example, movie_info_by_title or movies_info_by_actor). Specialized retrievers are added over time to handle question types that generic tools struggle with, improving precision and consistency.How does the Retriever Router choose and invoke tools?Typically an LLM with tool/function-calling support analyzes the question, selects the most suitable retriever(s), and extracts arguments (for example, capital_by_country("France")). The router may scale to many tools and more complex decision logic as your system grows.How are tool calls executed and what can’t the LLM do?The LLM proposes which tool to use and the arguments to pass, but your application executes the calls (for example, via a handle_tool_calls function). The LLM cannot directly run your retrievers; it only returns structured tool calls that your system must honor.What is Continuous Query Updating and why is it helpful?Questions are handled sequentially so previous answers can refine subsequent questions, making them atomic and specific. A query updater fills in missing details without asking for new information beyond the original question (for example, resolving “Who won the most Oscars, and is that person alive?” into two linked, updated questions).What does the Answer Critic do and how does it avoid loops?It inspects the collected answers to see if the original question is fully addressed; if not, it proposes complete, specific follow-up questions. You should set exit criteria to avoid infinite loops; in the example flow, answers are critiqued once before producing the final response, with the assistant explaining any remaining gaps.How is the full agentic RAG flow tied together?The system updates the user question (query_update), routes it to the right tool(s) (route_question), collects answers, critiques them (critique_answers), optionally gathers missing info, and finally produces an answer constrained to provided context (main). The chapter’s implementation uses OpenAI tool/function calling (e.g., GPT-4o), a Neo4j-backed Movies dataset, generic tools like text2cypher, specialized Cypher templates, and a simple “answer already in context” tool to avoid unnecessary retrieval.
pro $24.99 per month
access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
Essential GraphRAG ebook for free
