This chapter examines one of AI’s central limitations—hallucinations—framing them as outputs that are wrong, delivered with undue confidence, and liable to appear unpredictably. It distinguishes three common forms: fabricated facts that sound plausible, misinterpretations of existing information when context or intent is missed, and incorrect solutions to tasks that models assert as correct. The chapter also highlights two pervasive traits that make hallucinations especially risky: overconfidence, where systems defend mistaken answers, and sensitivity to small prompt changes that can flip results in surprising ways.
The underlying causes are traced to incomplete “world models,” long-tail edge cases that training data rarely covers, and a misalignment between what models are optimized to do (e.g., predict the next token) and what users want (truthful, reliable reasoning). A toy example in price optimization shows how models trained to predict purchases can “hallucinate” when repurposed to simulate pricing scenarios absent from the data. The chapter argues that simply scaling data may not solve the problem—usable data is finite, improvements may demand exponentially more examples, and current architectures (such as autoregressive language models with limited physical intuition and fixed inference steps) may be inherently prone to error, suggesting hallucinations are unlikely to vanish soon.
Given this persistence, the chapter surveys mitigation strategies: curating and cleansing training data, alignment techniques such as RLHF, training tweaks that better anchor facts, prompt engineering (including chain-of-thought), retrieval-augmented generation to ground responses, multi-agent setups where models critique each other, and uncertainty-based detection using token probabilities to flag dubious outputs. It closes with practical implications: hallucinations can create legal and safety liabilities (as seen in customer-service chat and autonomous driving), patch-by-patch fixes resemble “Whac-a-Mole,” and product teams should either target use cases where occasional errors are tolerable or invest early in mitigation and detection to manage risk.
An example of AI drawing what it is explicitly told not to.
An example of AI generating a surprising output. The output was different when asking the question as follows: “Which one is higher, 9.11 or 9.9?”
Summary
Hallucinations are confidently wrong outputs generated by AI.
Common types of hallucinations are made-up facts, misinterpreted information, and incorrect solutions to problems.
One cause of hallucinations is that AI’s internal “world model” is insufficient to describe how our world operates.
Another cause is that AI models are often trained to do one thing and used for something else—they’re “misaligned” with our goals.
Hallucinations are not going away anytime soon because this would require modifying prevailing machine learning methods in a yet-unknown way.
Hallucinations are sometimes unacceptable or unsafe for users, which can deeply hurt a product’s chances of success.
Hallucinations can be mitigated by using prompt engineering techniques, and they can sometimes be detected.
We must keep hallucinations in mind all throughout the lifecycle of an AI-related product.
FAQ
What are AI hallucinations?Hallucinations are unsatisfactory AI outputs with three key traits: (1) they’re incorrect (for example, a made-up fact or wrong solution), (2) they’re confidently presented as correct, and (3) they occur unpredictably, often where users least expect them.What types of hallucinations does the chapter describe?The chapter highlights three main types: (1) made-up facts (fabricated citations, titles, names), (2) misinterpretations of real sources (failing to grasp context or satire), and (3) incorrect solutions to tasks (logic, coding, or image-generation mistakes presented as correct).Can you give examples of “made-up facts” hallucinations?Yes. Notable cases include a legal brief with fabricated case citations produced by an LLM and an answer listing plausible-sounding but nonexistent book titles and authors—both presented as real.How do misinterpretations lead to hallucinations even when data exists?AI can pull text that exists online but fail to interpret it correctly. Examples include taking satire or forum posts at face value (e.g., “eat rocks” or harmful advice) and, in RAG systems, misunderstanding the very documents they’re given.Why do AIs sometimes give wrong answers with great confidence?Overconfidence is common: models often assert incorrect outputs as certain and may resist correction in follow-ups. Studies observe persuasive tone, reassurances of quality, and “stubbornness” even when the answer is wrong.What are the main causes of hallucinations?Key causes include: (1) inadequate world models learned from examples that miss real-world common sense and edge cases (e.g., cows only recognized on grass, unusual traffic cone patterns), (2) practical limits of data coverage for the “long tail” of rare scenarios, and (3) misaligned objectives—training to predict likely next tokens rather than to be truthful or task-correct.How does misaligned training objective contribute to hallucinations?LLMs are trained to predict the most probable next token, not to ensure factual accuracy or optimal problem-solving. Outputs that “look right” statistically can be wrong in reality. Even with RAG, a model may use retrieved text to sound more convincing rather than to ground truth reliably.Will hallucinations go away soon?Unlikely in the near term. Constraints include limited high-quality training data, diminishing returns (more data needed for smaller gains), and potential shortcomings of current autoregressive, text-only approaches. Some researchers argue that new methodologies may be required, and timelines for such breakthroughs are uncertain.What strategies can mitigate or detect hallucinations?Mitigations include: (1) curating/cleansing training data, (2) alignment methods like RLHF, (3) training tweaks (e.g., injecting contextual titles), (4) prompt engineering (e.g., chain-of-thought exemplars), (5) RAG to operate in an “open-book” setting, and (6) multi-agent setups where models critique each other. Detection can leverage low token-probability signals, which correlate with higher hallucination risk and warrant validation.How can hallucinations impact products and what should teams do?They can damage trust, trigger legal/PR issues, or kill products. Examples include an airline chatbot giving refund misinformation (leading to a court-mandated payout) and self-driving cars failing on edge cases despite patches. Teams should choose use cases tolerant to occasional errors, plan early for mitigation and detection, and avoid relying on models for counterfactual analysis beyond their training (e.g., naive price “what-ifs”).
pro $24.99 per month
access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
The AI Pocket Book ebook for free
