5 Detecting and quantifying hallucinations

This chapter explains that hallucination detection is fundamentally a factuality problem: fluent, confident text can still be unsupported, partially fabricated, or completely invented. Because standard metrics like BLEU, ROUGE, or even semantic similarity and generic judge models do not directly measure whether claims are grounded in evidence, the chapter frames hallucinations through a reliability lens. It separates output quality into two dimensions—contextual grounding and factual alignment—so that systems can distinguish faithful answers, intrinsic errors that misstate provided sources, extrinsic errors that add unsupported details, and mixed cases that look plausible but are still wrong.

The chapter’s centerpiece is SelfCheckGPT, a zero-resource method for open-domain detection that treats consistency across multiple stochastic generations as evidence of factuality. If a model truly knows a fact, it should reproduce it reliably; if it is fabricating, details tend to vary across samples. SelfCheckGPT operationalizes this by generating reference samples and comparing them to the target response using several variants, including BERTScore, NLI, question-answering, and LLM-as-a-Judge. The chapter also notes the method’s strengths and limitations: it works without external knowledge bases or model internals, but it can miss stable false beliefs, incurs extra compute, and is sensitive to sampling choices like temperature and number of samples.

For production use, the chapter recommends a layered pipeline: decompose responses into atomic claims, route source-backed claims to reference-based verification, use consistency-based methods for open-domain claims, and aggregate results into faithfulness scores and operational actions such as logging, review, or blocking. It emphasizes that verification is expensive compared with generation, so online systems often rely on lightweight signals such as token probabilities or small NLI models, while deeper checks are better suited to offline audits and regression testing. The broader takeaway is that hallucination detection should be integrated into evaluation and alignment loops, with thresholds calibrated to the application’s risk profile, and with the expectation that some hallucination will persist despite mitigation efforts.

A flow chart for choosing a hallucination detection strategy. Closed-domain tasks can use grounding based on sources to verify claims. Open-domain tasks require self-consistency checks, or rely on external fact retrieval, or other factual alignment signals.

A production hallucination detection pipeline. Generated responses are decomposed into claims, routed to the appropriate verifier, aggregated into response-level metrics, and converted into actions such as logging, review, regeneration, or blocking.

The LLM Reliability Quadrant. Contextual grounding (vertical axis) asks whether the output stays within the provided sources. Factual alignment (horizontal axis) asks if the output is correct according to verifiable evidence. The safest output is both grounded and aligned. The riskiest output is neither grounded nor aligned. The mixed-signal quadrants are operationally important because the output can look based on source but wrong, or look correct but not grounded.

An example of an agent assistant outputs answers in a very fluent and convincing manner, but the answer is incorrect.

The SelfCheckGPT pipeline. The same prompt produces one target response and N stochastic examples. Each sentence in the target is checked for consistency against the samples. Consistent sentences score as factual; inconsistent sentences score as hallucinated.

• Closed-domain tasks are constrained to a provided context (the 'source of truth') that the output must strictly adhere to. Open-domain tasks lack this explicit boundary, requiring the model to rely on its internal parametric knowledge or retrieve information from external sources like web search or a knowledge base.
• The LLM Reliability Quadrant classifies model outputs along two dimensions: contextual grounding (whether the output stays within the source) and factual alignment (whether it is correct). Intrinsic hallucinations are grounded but misaligned; extrinsic hallucinations are ungrounded and misaligned. The "extrinsic but correct" quadrant is deceptively dangerous because it signals unreliable generation even when the output happens to be right.
• SelfCheckGPT exploits the consistency principle: factual statements remain stable across multiple independent generations while hallucinations vary. This enables zero-resource detection without access to knowledge bases, model internals, or ground truth.
• SelfCheckGPT offers four variants with different trade-offs: BERTScore for fast, low-cost screening; NLI for detecting factual contradictions; QA-based for fine-grained claim verification; and LLM-as-a-Judge for the highest accuracy.
• SelfCheckGPT cannot detect consistent hallucinations where the model has stably learned incorrect information from training data. Complement consistency-based detection with reference-based verification whenever grounding sources are available.
• Claim decomposition and verification provide a structured alternative that breaks text into atomic factual claims and verifies each independently, producing granular, actionable faithfulness scores.
• Online hallucination detection prioritizes latency, using lightweight signals like token probabilities and fast NLI models. Offline detection prioritizes accuracy, using full SelfCheckGPT or comprehensive claim decomposition for testing and auditing.
• Hallucination detection integrates with the multi-dimensional evaluation framework from chapter 4 as a binary quality gate: responses that fail the faithfulness check should not proceed to graded evaluation of other dimensions.
• Detection is necessary but not sufficient. The mitigation hierarchy progresses from prompt engineering through retrieval augmentation, fine-tuning, and RLHF, with each level building on detection as the monitoring layer.
• Hallucination cannot be fully eliminated with current language model architectures. Design systems that assume some persistent hallucination rate and build human-in-the-loop safeguards for high-stakes decisions.

FAQ