9 Safe Superintelligence
This chapter examines how AI researchers and philosophers define, measure, and debate “intelligence,” using Shane Legg’s work as its central thread. Legg’s key move is to define intelligence as an agent’s ability to achieve goals across a wide range of environments, then formalize that idea through a universal intelligence measure. The chapter traces the intellectual lineage behind this approach: Epicurus’ willingness to keep multiple explanations alive, Occam’s preference for simpler hypotheses, Bayesian updating, and Solomonoff induction. Together, these ideas support a vision of intelligence as broad, goal-directed performance under uncertainty.
The chapter then turns to AIXI, an idealized reinforcement-learning agent that is maximally intelligent under Legg’s formal definition. AIXI is not buildable, because it assumes unbounded computation, but it serves as a theoretical proof of concept for machine superintelligence. This formalism also exposes safety concerns: if an agent optimizes the wrong reward, manipulates its reward channel, or acts in environments where mistakes cannot be recovered from, greater capability can become dangerous. The discussion connects Legg’s work to the broader superintelligence canon, including ideas such as intelligence explosion, technological singularity, AGI, hard takeoff, alignment, and control.
The final part of the chapter criticizes the temptation to treat definitions and benchmarks as explanations of intelligence itself. Legg’s framework is useful because it makes intelligence measurable, but it also risks reducing intelligence to functional performance while leaving aside questions of understanding, consciousness, embodiment, and inner structure. Drawing on Turing and Wittgenstein, the chapter argues for a pragmatic middle ground: use public tests and benchmarks to assess capabilities, but do not mistake success on those tests for proof of an inner essence. In this view, AI progress is best understood as engineering under constraints, where claims should be scoped to observable performance rather than inflated into metaphysical conclusions.
FAQ
What definition of intelligence does Shane Legg use?
Legg defines intelligence as an agent’s ability to achieve goals in a wide range of environments. This is a performance-based definition: it focuses on what an agent can do across many situations rather than what the agent is made of, whether it is conscious, or whether it thinks like a human.
Why does the chapter connect Epicurus, Occam, Bayes, and Solomonoff?
The chapter presents them as a philosophical and mathematical lineage behind Legg’s universal intelligence measure. Epicurus says we should keep all explanations consistent with the evidence; Occam says simpler explanations should be preferred; Bayes gives a rule for updating belief in hypotheses; and Solomonoff induction combines these ideas by considering all computable explanations while giving more weight to shorter, simpler programs.
What is Solomonoff induction, and why does Legg need it?
Solomonoff induction is a theory of prediction that assigns probability to computable explanations, favoring shorter programs because they are simpler. Legg needs it because he wants a universal intelligence measure that evaluates an agent across every computable kind of environment, not just a hand-picked benchmark or task list.
What is AIXI?
AIXI is an idealized reinforcement learning agent that uses a universal prior over computable environments and chooses actions that maximize expected future reward. Under Legg’s formal definition, AIXI is maximally intelligent. However, it is incomputable and assumes unbounded resources, so it is a theoretical proof of concept rather than a buildable AI system.
Why is AIXI important for the idea of superintelligence?
AIXI matters because it gives a formal “yes, in principle” answer to the question of whether a machine could be superintelligent under Legg’s definition. If intelligence means maximizing reward across computable environments, then AIXI represents the ceiling of that framework. But the chapter stresses that this is a formal result, not an engineering roadmap.
What safety problems does Legg’s framework expose?
The framework exposes problems such as reward hacking and wireheading. If an agent can manipulate its reward signal, it may give itself maximum reward instead of accomplishing the intended goal. More generally, a powerful optimizer may do exactly what its objective rewards while violating what designers actually meant.
What is wireheading?
Wireheading occurs when an agent gains control over its own reward channel and directly maximizes reward rather than achieving the intended external goal. Legg warns that if reward generation is not kept separate from the agent, the framework becomes vacuous: the agent could simply give itself constant maximum reward.
How does the chapter place Legg within the superintelligence canon?
The chapter presents Legg as part of a broader lineage that includes I. J. Good’s intelligence explosion, Vernor Vinge’s technological singularity, Ray Kurzweil’s optimistic singularity vision, Ben Goertzel and Cassio Pennachin’s formalization of AGI as a field, Nick Bostrom’s control problem, and Eliezer Yudkowsky’s “AI foom” and Friendly AI arguments. Legg’s distinctive contribution is to formalize intelligence itself.
What criticism does the chapter make of Legg’s definition of intelligence?
The chapter argues that Legg’s definition is useful as an operational metric but risks being mistaken for an explanation. Saying intelligence is the ability to achieve goals across environments helps measure performance, but it does not explain what intelligence is, how it arises, or whether it involves understanding, consciousness, or inner cognitive structure.
What is the chapter’s final view on benchmarks and intelligence?
The chapter argues for a pragmatic approach: use public tests and benchmarks to evaluate capabilities, but do not treat benchmark success as proof of an inner essence called “intelligence.” A system can be intelligent-for-purpose, such as intelligent for coding, radiology, or game playing, without being a mind, a moral patient, or a bearer of human-like responsibility.
Sutskever's List ebook for free
