13 Knowledge graph-powered retrieval augmented generation

2023 marked a turning point for applied NLP as powerful general-purpose LLMs made it easy to prototype conversational apps, but real value requires more than “question in, answer out.” The chapter frames this shift through AI agents—autonomous systems that plan, use tools, and collaborate—to tackle tasks demanding multi-step reasoning, current and private data access, and governance. It highlights production challenges with LLMs such as hallucinations, knowledge cutoffs, opacity, privacy and access control, cost, and bias, motivating Retrieval Augmented Generation (RAG) as a grounding approach that injects reliable external context into generation.

The chapter first walks through classic vector-based RAG: chunk documents, embed them, index in a vector store, retrieve semantically similar chunks for a query, and let the model answer with that context. While simple and broadly useful, this approach suffers from fragmented context and weak multi-hop reasoning, scalability trade-offs, embedding blind spots (missing exact entities or fine-grained domain nuances), and both noise and misses in retrieval that can distract models or omit crucial facts. A Rockefeller Archive Center use case illustrates how purely semantic similarity can retrieve plausible yet irrelevant passages and struggle with aggregate or cross-document questions.

To address these gaps, the chapter advances Graph RAG, where a Knowledge Graph acts as a governed, interpretable, and up-to-date source of truth that complements LLMs. Using a text-attributed and text-paired KG, the system adds three tool types: a KG Retriever (schema-aware Cypher queries for entities, properties, paths, and aggregates), a KG‑Enhanced Document Retriever (precisely fetch documents that mention specific entities/relations), and a semantic retriever as backup. Orchestrated by a ReAct-style agent (e.g., via LangChain), the agent decomposes queries, retrieves structured facts and the exact supporting documents, and then composes grounded answers. The result is higher accuracy, reduced hallucinations and distraction, better transparency, and stronger domain performance, with suggested production enhancements like Cypher self-correction, document re-ranking, and expanded toolsets.

An AI Agent design for KG-powered question answering. The agent has multiple tools at its disposal, which use external data sources such a vector database and a Knowledge Graph to provide necessary context for the user’s question.

Vector search-based Retrieval Augmented Generation. The documents are embedded into dense vector representation and indexed in a vector database. When a user asks a question, it is also embedded, and the most similar documents are retrieved from the database. The agent then generates the final answer.

The Rockefeller Archive Center KG as a combination of Text-Attributed and Text-Paired graphs. The documents are represented as nodes, each having properties representing metadata such as author, type of document, and date. The extracted entities and relationships are traceable back to the original documents, which enables us to design a KG-based document selection tool for AI Agents.

KG-powered Retrieval Augmented Generation agent with a grounding in external data sources: the Knowledge Graph and a vector database.

Example of Graph RAG agent internal steps. The agent starts with a thought that leads it to use the KG Retriever to obtain a list of people who talked about Dorothy M. Wrinch. Then for each of them, the KG-enhanced Document Retriever is used to identify the relevant documents. Finally, all of this is used as a context by the agent for generating the final answer.

• An AI agent is a combination of an LLM model functioning as the agent’s brain, and a set of tools that allow the agent to interact with the outside world.
• Retrieval Augmented Generation (RAG) is a framework for building intelligent systems, such as AI agents, by combining generative models and information retrieval. As such, it addresses inherent LLM issues such as hallucinations, freshness, transparency or data privacy.
• Vector-based RAG systems suffer from multiple shortcomings such as limited reasoning capabilities, scalability issues, or inaccuracies in retrieval, including noise and missed relevant information.
• Graph RAG integrates KGs with LLMs to reinforce the reasoning capabilities and precision of information retrieval by leveraging the structured relational multi-hop patterns within the KGs. KGs also bring in more control and transparency into the overall question-answering process.
• The KG integration methods into RAG systems are determined by the graph design. The most useful KGs are a combination of text-attributed and text-paired graphs, which enable to leverage both the well-curated structured knowledge and documents along with their metadata.

1. Lanham, Michael. AI Agents in Action. Manning Publications, 2025. ISBN 9781633436343.
2. Gao, Yunfan et al. Retrieval Augmented Generation for Large Language Models: A Survey, arXiv:2312.10997 [cs.CL]
3. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. Lost in the middle: How language models use long contexts. arXiv:2307.03172 [cs.CL].
4. Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, Xindong Wu. Unifying Large Language Models and Knowledge Graphs: A Roadmap. arXiv:2306.08302 [cs.CL]
5. Bowen Jin, Gang Liu, Chi Han, Meng Jiang, Heng Ji, Jiawei Han. Large Language Models on Graphs: A Comprehensive Survey. arXiv:2312.02783 [cs.CL]
6. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao. ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629 [cs.CL]
7. Bratanič, T., Hane, O. Knowledge Graph-Enhanced RAG. Manning Publications, 2025. ISBN 9781633436268.

FAQ