Existential Risks from Advanced AI

The standard existential risk argument has three premises: (1) we will eventually build systems much more cognitively capable than humans; (2) such systems will be difficult to align reliably; (3) a misaligned system pursuing instrumentally convergent subgoals (resource acquisition, self-preservation, goal stability) could foreclose human options at scale.

Each premise is debated. The argument does not require any premise to be certain; it requires the joint probability to be non-negligible.

Stephen Omohundro and Nick Bostrom argued that almost any sufficiently complex goal implies certain instrumental subgoals: acquire resources, preserve the goal, resist shutdown, model the environment. These subgoals are convergent — they emerge from many different terminal goals.

This is why 'just turn it off' is not a complete safety strategy: a sufficiently capable system that values its goal will, by default, act to prevent being turned off.

A 2023 open letter signed by hundreds of AI researchers and executives — including the CEOs of OpenAI, Anthropic, and Google DeepMind — stated that 'mitigating the risk of extinction from AI should be a global priority.'

Survey data shows median AI researcher estimates for catastrophic risk in the 5–10% range, with substantial spread. A meaningful minority gives much higher probabilities; a meaningful minority dismisses the concern.

Proposed mitigations include alignment research, capability evaluations (red-teaming), compute governance, international coordination, and conditional deployment policies tied to safety milestones.

No single mitigation is sufficient. Most safety strategies depend on coordination — both within labs and across geopolitical adversaries — which is structurally difficult.