6 Constructing knowledge graphs with LLMs
This chapter explains how to transform unstructured documents into reliable knowledge graphs using large language models, focusing on when simple vector retrieval is not enough. It motivates the need for structured data to answer precise, business-style questions that require filtering, counting, and aggregation, highlighting common pitfalls in domains like legal contracts where naive chunk-and-retrieve can mix contexts across documents. The chapter lays out an end-to-end workflow: define what to extract, use LLMs to produce structured outputs, and load the results into a graph database for accurate, composable querying.
The core technique is structured data extraction guided by explicit schemas. Using model-constrained outputs, developers specify fields, types, enums, and descriptions to steer extraction, balancing mandatory versus optional attributes to limit hallucination and nesting objects only where it adds clarity. A contract model (with parties, dates, terms, scope, amounts, and governing law) demonstrates how to express domain intent, pair it with targeted instructions, and obtain consistent JSON-like results. A public contract example shows how LLMs can populate the schema from raw text, producing usable, standardized records even when some fields are missing.
With structured records in hand, the chapter shows how to design and populate a Neo4j graph with Contract, Organization, and Location nodes and their relationships, supported by constraints and indexes for integrity and performance. It discusses practical loading considerations (such as idempotency), then turns to post-load refinement, emphasizing entity resolution to merge duplicates arising from naming variations. Finally, it advocates combining structured facts with retained unstructured text, using domain-aware chunking (for example, by clauses in contracts) to preserve context. The result is a graph that supports precise analytics and richer retrieval than embeddings alone.
Basic vector retrieval strategy might return chunks from various contracts
Building knowledge graphs from text by using LLMs to extract structured data information
Contract graph model
Contract graph data visualized
Potential duplicates
Expanded graph model with added unstructed data
Summary
- Simply chunking documents for retrieval can result in inaccurate or mixed results, especially in domains like legal documents where document boundaries matter.
- Retrieval tasks like filtering, sorting, and aggregating require structured data, as text embeddings alone are not suited for such operations.
- LLMs are effective at extracting structured data from unstructured text, converting it into usable formats like tables or key-value pairs.
- Structured output features in LLMs allow developers to define schemas, ensuring responses follow a specific format and reducing the need for post-processing.
- Defining a clear data model with attributes such as contract type, parties, and dates is essential for guiding LLMs to extract relevant information accurately.
- Entity resolution in knowledge graphs is important for merging different representations of the same entity, improving data consistency and accuracy.
- Combining structured and unstructured data in knowledge graphs preserves the richness of the source material while enabling more precise querying.
FAQ
Why are plain text embeddings insufficient for answering many questions over legal contracts?
Because vector search returns semantically similar chunks without guaranteeing document provenance or supporting operations like filtering, counting, and aggregation. In multi-contract collections, top-K chunks can mix terms from different contracts, producing confused answers. Structured extraction ties facts to the correct contract and enables BI-like queries (filter, group, count).What is structured data extraction with LLMs and why use it?
It’s the process of converting unstructured text into a predefined schema (for example, JSON). LLMs can be guided to identify entities, attributes, and relationships, producing consistent, machine-usable outputs that power knowledge graphs and enable precise querying beyond similarity search.How do OpenAI Structured Outputs and Pydantic help ensure reliable formats?
Define a schema (for example, with Pydantic classes) and pass it as the response format. The model is constrained to that schema, providing type safety and reducing post-processing. Use attribute descriptions, enums, and clear names to guide extraction and minimize ambiguity.Which fields belong in a basic contract data model, and which should be optional?
Typical fields: contract_type (enum), parties (list of Organization with name, location, role), effective_date, term, contract_scope, end_date (optional), total_amount (optional), governing_law (Location). Mark fields optional when they may be missing to avoid hallucinated values.How should I handle dates, locations, and controlled vocabularies in the schema?
- Use strings for dates and specify the exact format (for example, yyyy-MM-dd) in the description.- Represent locations with a nested Location object and standardize country as a two-letter ISO code.
- Use enums where appropriate (for example, contract_type) to constrain values.
What does an end-to-end extraction request look like?
- Define a focused system message (domain, task, output use).- Send the raw document as the user message.
- Use a model that supports structured outputs, set temperature low (for example, 0), and provide the schema as response_format.
- Parse the result into JSON aligned to your data model.
What dataset is used in the chapter’s example and why?
The CUAD (Contract Understanding Atticus Dataset) is used because proprietary contracts are often confidential. The book’s repository includes a sample contract to demonstrate extraction without downloading the entire dataset.How do I import the extracted data into Neo4j?
- Design a graph model (for example, Contract —HAS_PARTY→ Organization —LOCATED_AT→ Location).- Create unique constraints and indexes for integrity and performance.
- Write a Cypher import query that MERGEs nodes and relationships and sets properties.
- Avoid non-idempotent IDs (for example, randomUUID) if you plan to re-run imports.
Essential GraphRAG ebook for free
