9 When machines seem to think

The chapter argues that artificial intelligence cannot be judged by failure alone or by impressive output alone. Human cognition is also fallible, so hallucinations, brittleness, and bias do not prove the absence of intelligence; at the same time, fluent explanations, coding, translation, and problem solving do not automatically prove understanding. Instead, the text asks how much observable performance should count, what role mechanism should play, and whether current systems belong outside intelligence or inside it in a form unlike our own.

To address that question, the chapter explores several ways machine behavior can resemble thought. It distinguishes brittle imitation from adaptive imitation, showing that language models have moved beyond simple pattern matching in many tasks. It then examines chain-of-thought and reasoning models, where intermediate steps improve performance by giving the system more structure to work with. The discussion of emergence adds that abilities can appear at scale from the interaction of many simple parts, making it hard to dismiss machine competence just because the underlying mechanism is token prediction.

The skeptical case remains strong on several fronts: language models can lack grounding, embodied experience, stable agency, enduring selfhood, and human-like learning. They may also fail on controlled reasoning benchmarks and still produce confident falsehoods. But the chapter concludes that these shortcomings do not erase the broader pattern of flexible competence. Current AI systems seem best described as genuinely intelligent in a limited, qualified sense: not like humans, not complete minds, but systems whose cognition is real, uneven, and strange.

Brittle and adaptive imitation. A brittle system may appear capable when tested on familiar cases, but its performance collapses when the situation changes. In chess, this would be a system that plays well only in familiar positions. An adaptive system continues to perform in unfamiliar cases, making the difference between imitation and ability harder to draw.

Intelligence inferred from observable signs. Standard definitions describe intelligence through capacities such as deriving information, learning from experience, adapting to the environment, and using reason. These capacities are recognized through performance, while the underlying mechanism remains partly hidden.

Same tools, different act. A master copyist may use the same tools and produce a visually similar result, yet intention, authorship, history, and context still shape how we judge the work. The example shows why skilled performance can be real without settling what kind of act the performance belongs to.

Direct answer versus chain of thought. A direct answer can jump to a plausible but incorrect result, while a chain of thought decomposes the problem into intermediate steps that provide structured context for the final answer.

Reasoning models and hidden intermediate work. Reasoning models often present the user with a familiar “thinking” state while additional computation happens before the answer. The intermediate process may involve generated steps, checks, and revisions, but it is usually hidden, compressed, or only partly reflected in the final response.

Emergence as system-level behavior. A single neuron does not perceive, remember, or think, and a single bee does not organize the colony. Yet through interaction at scale, neurons can give rise to cognition and bees can produce coordinated collective behavior. In both cases, the emergent property depends on the components but becomes visible only through their organization and interaction.

Grokking as delayed generalization. Training loss falls early as the model fits the examples, but validation loss remains high until much later, when the model begins to generalize to new cases.

Three performance regimes reported in “The Illusion of Thinking.” Standard language models perform well at low complexity, while reasoning models gain an advantage at intermediate complexity. At high complexity, both decline sharply, and reasoning models may reduce their reasoning effort even when additional token budget remains available.

The stochastic parrot analogy. Parrots may reproduce human speech without human-like understanding, while language models generate new text from statistically learned patterns without direct grounding in what the words describe.

Embodied interaction and cognition. Through repeated interaction and consequences, an agent can develop a world model, exercise agency, and represent itself within the environment. These capacities remain closely connected, each shaping and constraining the others.

Different learning paths. A child does not need to see millions of glasses fall to learn that unsupported objects drop and fragile ones may break. Human learning is cyclical and ongoing, whereas AI systems usually acquire comparable regularities during training and then remain largely frozen after deployment.

We often infer intelligence through communication before knowing the body or mechanism behind it. If an extraterrestrial source produced memory, reasoning, and problem solving through signals alone, embodiment would not be the first condition we demanded.

Intelligence need not depend only on direct bodily experience. Humans acquire much of their knowledge through symbols, records, instruments, testimony, and education, while artificial systems can be connected to the world through sensors, actions, and feedback without sharing our biological form.

Agency and intelligence. Humans usually combine agency and intelligence, making the two easy to confuse even though they are different capacities. Agency concerns pursuing goals as one’s own; intelligence concerns understanding, reasoning, planning, and solving problems.

• AI systems challenge the duck-test intuition because they display many outward signs of intelligence while being produced by mechanisms unlike perception, intention, or lived experience.
• Brittle imitation collapses when conditions change, while adaptive imitation remains flexible across unfamiliar cases, making the boundary between imitation and ability harder to draw.
• Large language models occupy a difficult middle ground between adaptive imitation and intelligence, because their performance is flexible enough to look like ability while their mechanism remains unfamiliar.
• Intelligence is not directly observable, but inferred from patterns of learning, adaptation, reasoning, problem solving, and performance.
• Humans adjust their thresholds for intelligence across children, adults, animals, contexts, and capacities, showing that intelligence is judged relative to the system being evaluated.
• Mechanism helps distinguish performance from competence, but when performance becomes flexible, repeatable, and context-sensitive, mechanism alone cannot settle the judgment.
• Chain-of-thought prompting and reasoning models show that reasoning-like behavior may depend on how a model is allowed to answer and how much intermediate structure it can build.
• Emergent abilities show that simple local mechanisms can produce surprising system-level behavior when learned relationships interact at scale.
• Prompts, tools, longer context, chain-of-thought, and reasoning models can act as amplifiers that reveal abilities not visible in ordinary interaction.
• The “illusion of thinking” critique shows that reasoning models can collapse under controlled increases in complexity, especially when stable procedural execution is required.
• Failures in reasoning benchmarks may reflect reasoning limits, execution limits, interface limits, or benchmark flaws, making evaluation of machine cognition difficult.
• The stochastic parrot critique argues that language models may reproduce statistically learned patterns without grounding, originality, or genuine understanding.
• Hallucinations reveal serious failures of reliability and grounding, but unreliable cognition is not the same as absent cognition.
• Embodiment critiques argue that intelligence may require bodily interaction, agency, world models, and a lived position from which the world matters.
• Disembodiment does not automatically exclude intelligence, because humans also acquire much knowledge through symbols, instruments, testimony, and other mediated routes.
• Agency, selfhood, and intelligence are different capacities, even though they usually appear together in humans.
• Language models learn differently from humans, acquiring most competence through data-hungry training before deployment rather than through continuous lifetime experience.
• Current AI systems should be considered intelligent only in a qualified sense: intelligence with missing pieces, strange strengths, and unfamiliar limits.

FAQ