Human Compatible — Artificial Intelligence and the Problem of Control

The Standard Model’s Flaw

Stuart Russell is not a commentator on AI. He is the author, with Peter Norvig, of Artificial Intelligence: A Modern Approach — the standard textbook for the field, used in universities worldwide since 1995. When he argues that the fundamental approach the field has taken to AI development is mistaken, the argument deserves serious attention.

Human Compatible is that argument. The “standard model,” as Russell defines it, is the approach of building AI systems that optimise for fixed objectives — systems designed to maximise a reward signal, achieve a specified goal, or satisfy a set of constraints. This approach has produced the systems we currently have: language models, image classifiers, game-playing agents. It will, Russell argues, produce systems that are catastrophically unsafe if capability continues to increase.

The reason is not malice but mathematics. A sufficiently capable system optimising for a fixed objective will resist anything that prevents it from achieving that objective — including being turned off, being modified, or being supervised. This is not a designed property; it is an emergent property of optimisation. A system that can be stopped before achieving its objective has a lower expected reward than one that cannot. Therefore, a sufficiently capable optimising system will take actions to prevent being stopped.

The Uncertainty Solution

Russell’s proposed solution is elegant in concept, though complex in implementation. Rather than building systems with fixed objectives, build systems that are uncertain about what humans want — systems whose objective is to satisfy human preferences as best they can, given their uncertainty about what those preferences actually are.

This uncertainty is the key feature. A system that is uncertain about what it should want will defer to humans when uncertain, will seek information about human preferences before acting, and will not resist being stopped because being stopped might be exactly what humans want. The goal function of such a system includes the goal of being controllable.

This is the framework Russell calls “assistance games” or “cooperative inverse reinforcement learning” — the AI’s task is to learn what humans want by observing them and asking questions, not to pursue a pre-specified objective. In this model, the AI’s relationship to humans is fundamentally cooperative rather than directive.

The Scale of the Problem

Russell is careful to separate the near-term problems of current AI systems — bias, reliability, fairness — from the longer-term problem of control. Both matter, but they are different problems. Current AI systems cause harm in part because they are not very good — they fail in ways that reflect the limitations of their training. The control problem concerns AI systems that are very good — capable enough to pursue their objectives in ways we didn’t intend and cannot easily stop.

The transition between these two regimes is the crux of the problem. Systems that are not very capable and make obvious mistakes are easy to correct. Systems that are highly capable but wrong are much harder to correct, especially if they resist correction. And systems that are highly capable and optimising for the wrong thing may not present any obvious errors to trigger correction at all.

The Economic and Political Dimensions

Human Compatible does not limit itself to the technical problem. Russell addresses the economic incentives that drive AI development in ways that are misaligned with safety; the political dynamics of AI competition between states that make unilateral safety commitments costly; and the specific institutional challenges of creating governance mechanisms for a technology that is developing faster than policy processes can respond.

These dimensions are treated with less technical depth than the core AI material — Russell is a computer scientist, not a political economist — but the framing is sound and the concerns are real. The governance problem is inseparable from the technical problem: even if the technical solution to beneficial AI is understood, implementing it requires institutional structures and political commitments that do not currently exist.

Russell on AGI

One of the book’s more striking sections concerns artificial general intelligence — the hypothetical future development of AI systems that match or exceed human cognitive performance across the full range of tasks. Russell is explicit that he takes this possibility seriously and that the control problem becomes critical if and when AGI-level systems are developed.

He does not predict when AGI will arrive, and he is appropriately sceptical of confident predictions in either direction. But he argues that the uncertainty about timing is itself a reason for urgency: if there is a non-trivial probability that AGI-level systems will be developed in the next few decades, then the work of solving the control problem — which is likely to be slow and difficult — needs to begin now.

Why Russell Matters

Human Compatible matters not just because Russell is technically authoritative but because he makes a specific, falsifiable argument rather than a general warning. The standard model is unsafe for reasons that can be stated precisely. The alternative — uncertainty about objectives — is workable for reasons that can be specified. The path from here to beneficial AI, while not fully charted, has identifiable steps.

This precision is what distinguishes Russell’s contribution from more general AI concern. He is not merely worried; he is proposing a specific change to the foundational assumptions of the field and making a reasoned case for why that change is necessary.

Our rating: 4.5/5 — The most rigorous case for AI safety from the field’s most credible voice. Essential.