5 Domain-specific knowledge extraction from unstructured data

The chapter explains how to unlock knowledge from domain-specific unstructured text and shape it into connected, queryable knowledge graphs. It frames the task as a multistep pipeline—ingestion, NLP, enrichment, modeling—and highlights two overarching challenges: knowledge representation (how to model information so it is reusable and efficient) and knowledge learning/construction (how to mine insights from text). Using the Rockefeller Archive Center as a case study—particularly program officers’ diaries rich in domain jargon, abbreviations, and historical context—the chapter shows why bespoke extraction is needed to reveal patterns in grant-making, influence networks, and evolving research trends.

The core technical focus is on named entity recognition and relation extraction as the backbone of building KGs from text. Generic off-the-shelf models are a start but insufficient for domain nuances such as fine-grained research topics, historical titles, and abbreviated references (“U. Cal.”, initials-only people). The chapter contrasts traditional NLP pipelines with Large Language Models, showing how LLMs can collapse multiple steps (NER, coreference, normalization) thanks to transfer learning and generative reasoning—while cautioning about hallucinations and output instability. A practical, iterative prompt-engineering approach is presented: clearly define target entity and relation classes, provide representative examples, tune parameters (e.g., temperature 0 for determinism), and enrich outputs with properties (e.g., sentiment on “talked about”) for downstream analysis. The authors stress rapid prototyping, mini-KG “eyeballing,” unit-style prompt tests, and quantitative QA to stabilize outputs; when prompts plateau, fine-tuning the model with a small curated dataset can deliver the needed accuracy and consistency.

Finally, the chapter weighs when to choose traditional NLP versus LLMs. Traditional approaches offer speed, simpler and cheaper infrastructure, and easier on-prem security, but demand significant expertise, multiple custom models, and costly annotation. LLMs provide a faster path to domain adaptation, an all-in-one extraction capability with strong language understanding, and reduced post-processing, but can be slower, more expensive to serve, harder to deploy securely at scale, and may require fine-tuning for mission-critical quality. The guidance is pragmatic: start with prompt-based LLMs for exploration and rapid value, measure and harden with disciplined evaluation, fine-tune when needed, and remember that both toolsets have a place—LLMs can even pre-annotate data to accelerate traditional pipelines. The end goal is accurate, normalized, and connected knowledge that supports complex querying over information previously locked in unstructured documents.

Path from domain-specific unstructured textual data towards structured knowledge. Each of the key steps relies on state-of-the-art Machine Learning models, be it for example Optical Character Recognition for document digitization, NER or Relation Extraction system.

A snippet of Warren Weaver's diary entry with highlighted relevant named entities.

Mental model of transforming textual data into structured knowledge: Named Entity Recognition and Relation extraction using a Large Language Model.

The two roles of Large Language Models in KG generation from textual data: prompt-based approach (upper branch of the flow) vs. fine-tuning approach (lower branch).

• Producing Knowledge Graphs from unstructured textual data requires the identification of custom domain-specific named entities and relations among them.
• A whole range of traditional NLP models, such as Named Entity Recognition, Relation Extraction, or Coreference Resolution, have to be trained and chained in a complex workflow. Training each of them requires a high initial investment for building high-quality human-annotated training datasets, and specialized data science skills to train, evaluate, deploy, and monitor the models.
• Large Language Models, such as OpenAI’s GPT, can be used out-of-the-box with iterative prompt engineering (or alternatively fine-tuning) to build domain-specific Knowledge Graphs of high accuracy with much lower initial costs compared to the traditional NLP.
• Traditional NLP and LLMs both have their pros and cons, depending on the concrete domain and various requirements and business considerations. They can even coexist side by side, or one can be used to prepare a training dataset for the other.
• The generative nature of LLMs means that we get cleaner more accurate KG with minimized need for entity normalization and even resolution.

1. Chernow, Ron (May 5, 1998). Titan: “The Life of John D. Rockefeller Sr.,” New York: Random House. pp. 563–566. ISBN 978-0679438083.
2. A. Vaswani et al., “Attention Is All You Need,” 2017, arXiv:1706.03762 [cs.CL]
3. J. Kaplan et al., “Scaling Laws for Neural Language Models,” 2020, arXiv:2001.08361 [cs.LG]

FAQ