6 The reliability problem

The chapter explains that large language models can sound highly reliable while still producing answers that are only loosely tied to truth. Their outputs are generated by predicting likely text continuations from patterns in training data and current context, not by checking facts against a stable internal record of reality. This makes hallucination a structural risk: the model may remain fluent, detailed, and confident even when it lacks solid evidence, which can mislead users into trusting outputs that are not actually grounded.

It also shows that reliability problems are amplified by the way models are trained and prompted. Because training rewards plausible completion more than refusal, models are encouraged to answer even when uncertain, and weak or ambiguous prompts can push them into overgeneralization, fabrication, or jailbreak-like behavior. Small changes in wording, tone, formatting, or context can shift outputs in surprising ways, revealing that instruction-following is a statistical pattern rather than true rule enforcement. As a result, models can appear to break rules or become inconsistent whenever contextual cues compete or when the prompt pushes them beyond familiar territory.

The chapter concludes that reliability must be engineered around the model rather than assumed inside it. External context, retrieval-augmented generation, chain-of-thought and verification steps, uncertainty monitoring, filtering, and human oversight can all reduce errors, but none can eliminate them entirely. Because the model’s active context is limited and attention is uneven, even relevant information may be underused or misplaced, and retrieved evidence does not automatically guarantee grounding. The central lesson is that language models can be useful within carefully designed boundaries, but their outputs remain probabilistic and need validation when accuracy matters.

Models learn patterns from training data and extend them to new cases. In this example, animals associated with water are often labeled as fish. When no clear example is available, as with the dolphin, the model does not stop or admit uncertainty. It continues by extending the strongest available pattern, incorrectly classifying the dolphin as a fish rather than a mammal.

A model trained within a limited range learns a trend and generalizes by extending it beyond the observed data. When applied outside that range, the same mechanism produces confident yet unrealistic predictions. Hallucinations in language models arise in an analogous way, as extrapolation continues under insufficient constraint.

Reducing hallucination through external context. When model knowledge is incomplete or unreliable, relevant information can be introduced at generation time through the prompt. By incorporating external data, the model’s output is guided toward available evidence, reducing unsupported claims and increasing the likelihood of producing a grounded answer.

Structuring generation through reasoning and verification. In a direct approach, the model produces an answer in a single step. Chain-of-thought (CoT) prompting introduces intermediate reasoning steps that make the path to the answer explicit. Chain-of-verification (CoV) extends this process by adding a second stage in which the initial answer is checked against supporting conditions, reducing the likelihood of unsupported conclusions.

Uncertainty in token prediction. When a model assigns high probability to a single continuation, generation is stable and strongly constrained. When multiple alternatives have similar probabilities, the model operates under higher uncertainty, increasing the likelihood of unsupported or fabricated outputs.

Language generation as probabilistic continuation. Given a prompt, the model produces a probability distribution over possible next tokens based on patterns learned during training. The response emerges by selecting tokens from this distribution step by step, rather than by executing explicit rules.

Positional influence within the context window. Information placed near the beginning or end of the context tends to influence the model’s response more strongly than information located in the middle, illustrating that presence in the context window does not guarantee equal impact on the output.

Retrieval-augmented generation pipeline. A user query is transformed into an embedding, a vector representation used for similarity comparison against embedded documents in a knowledge base. The system ranks documents by semantic similarity and inserts only the top matches as text into the prompt before generation. The model’s response is thus conditioned on both its internal parameters and selectively retrieved external knowledge.

• Hallucinations arise when language models generate outputs that appear coherent and confident but lack grounding in verifiable evidence, emerging from the statistical nature of language modeling, especially when relevant knowledge is sparse or ambiguous.
• Structuring generation through intermediate reasoning and verification steps, such as chain-of-thought and chain-of-verification approaches, can improve performance by making assumptions explicit and reducing unsupported conclusions.
• Monitoring uncertainty during generation can help identify outputs that may be unreliable, using internal signals as indicators of weak statistical support rather than as guarantees of correctness.
• Providing reliable external information through the prompt can reduce hallucinations by grounding responses in identifiable sources, though no technique fully eliminates them within a purely predictive framework
• Model behavior is governed by statistical regularities learned during training rather than by explicit rules or hard-coded directives.
• Because generation follows probabilistic patterns, small differences in wording can shift responses substantially, and no universal prompt formulation guarantees optimal results.
• Repeating or restructuring input within the prompt can reinforce the influence of relevant information, partially counteracting positional effects in the context window.
• Safety alignment functions as a probabilistic influence rather than a categorical barrier, shaping model tendencies without enforcing absolute constraints.
• Jailbreaks exploit the probabilistic nature of alignment by crafting prompts that shift the statistical balance toward patterns associated with compliance rather than refusal.
• Various safeguards attempt to reduce undesirable behavior, including input filtering, output moderation, architectural constraints, structured prompting, and external monitoring.
• Language models can only reason over information contained in their internal parameters and the material present within the current context window.
• Even within that window, attention mechanisms distribute influence unevenly, so some information may dominate the response while other relevant signals are partially ignored.
• Retrieval-augmented generation addresses the limited capacity of the context window by selectively injecting external knowledge into the model’s working context.

FAQ