6 Building knowledge graphs with large language models

This chapter shows how to move from language-model-based knowledge extraction to a usable, explainable Knowledge Graph. Using the Rockefeller Archive Center diaries as a running example, it tackles real-world obstacles: converting analog, historical, idiosyncratically written documents via OCR; handling domain-specific entities of uneven granularity; coping with dense relational content; and performing the post-extraction heavy lifting of normalization, cleansing, entity resolution, and cross-source reconciliation. The emphasis is on why LLMs make this tractable compared to traditional NLP, while also making clear that disciplined graph-building practices remain essential.

The authors propose a three-layer graph design: a document layer to preserve provenance (files and pages), a meta-graph layer of raw entity mentions and relations from the LLM, and a final KG layer of resolved entities (persons, organizations, occupations, titles) and normalized relations. Pages are processed as chunks; mentions and their relationships are stored without merging, then cleaned (for example, lowercasing occupations and stripping honorifics from names) and resolved. Although LLMs implicitly help with coreference, robust entity resolution still relies on graph signals: string similarity rules informed by domain conventions, corroborating relationships such as WORKS_FOR and WORKS_ON, and community structure. Weakly connected components unify mentions into canonical nodes, while semantic embeddings cluster occupation terms to bridge vocabulary gaps and granularity differences—all while preserving traceability back to source text.

With a consolidated KG, graph data science reveals an “intellectual network” of scientists via relations like TALKED_WITH, TALKED_ABOUT, WORKS_WITH, and STUDENT_OF. Centrality metrics highlight influencers, bridges, and key connectors; focused multi-hop queries surface people and organizations around topics such as cyclotron research. The chapter also stresses feedback loops to catch extraction errors, then outlines scalable next steps: improving prompts or fine-tuning, handling multi-page context, resolving occupations at multiple levels, linking officer diaries to grants, and modeling conversations and their chains. Finally, it argues for the enduring value of KGs in the LLM era: they enable explainability and provenance, “de-blackbox” model outputs, democratize expensive models by extracting once and reusing often, foster exploratory analysis, and provide precise, controllable analytics beyond what a monolithic language model alone can reliably offer.

Path from domain-specific unstructured textual data towards KG insights. Each of the key steps relies on state-of-the-art Machine Learning models, be it for example Optical Character Recognition for document digitization, NER and Relation Extraction systems, Entity Resolution and Graph ML.

Simplified Knowledge Graph schema of the Rockefeller Archive Center project.

Graph-based Entity Resolution example.

Intellectual influence network made of relations TALKED_ABOUT, TALKED_WITH, WORKS_WITH and STUDENT_OF. Node styling is based on their betweenness centrality score.

Influence network in the 2-hop vicinity of the cyclotron research. The person node sizes are based on global PageRank centrality to highlight popular nodes in the graph.

Physics research influence network connecting Johns Hopkins University and Harvard University with no more than three hops among persons. The green and red relationships indicate positive and negative sentiments.

• Knowledge Graph schema design, a.k.a. graph modelling, is a key step that ensures the information is stored in the most optimal way for the specific use case at hand. Well-designed schema simplifies the KG creation, entity, and relation cleansing, normalization, and resolution, as well as ensures efficient downstream analyses and insights discovery optimized for the specific use case.
• Extracting high-quality entity relations from textual documents helps to design an unsupervised graph-based entity resolution approach.
• Various graph data science and graph Machine Learning techniques help to analyze patterns and draw insights from Knowledge Graphs, which serve as an intermediate step towards answering complex questions without relying too heavily on black-box tools such as LLMs.

FAQ