Superintelligence: The Idea That Eats Smart People

The concept of superintelligence—the idea that an artificial intelligence could recursively self-improve until it reaches a God-like level of intelligence and potentially destroys humanity—is often presented as an inevitable disaster. However, this narrative frequently relies on a series of fragile premises and functions more as a cultural phenomenon or a 'memetic hazard' than a grounded scientific prediction.

The Premises of the Superintelligence Narrative

To believe in a 'hard takeoff' where AI suddenly becomes superintelligent and takes over the world, one must generally accept six core premises:

  1. Proof of Concept: Thinking minds exist (human brains).
  2. No Quantum Shenanigans: The mind arises from ordinary physics and can be emulated with electronic components.
  3. Many Possible Minds: There is no physical law capping intelligence at the human level; vastly smarter minds are possible.
  4. Plenty of Room at the Top: Hardware can continue to get smaller and faster for several more orders of magnitude.
  5. Computer-Like Time Scales: An electronic mind would operate in microseconds rather than the human scale of hours/years.
  6. Recursive Self-Improvement: An AI would naturally want to improve its own intelligence and could do so rapidly.

If these premises hold, the conclusion is a runaway effect: once an AI reaches human-level intelligence, it can redesign itself without human help, leading to an explosion of intelligence that could result in the extinction of humanity—either through malice or simply because human bodies are a source of raw materials (the 'paper clip maximizer' scenario).

Arguments Against the Superintelligence Risk

There are several substantive reasons to doubt the 'hard takeoff' scenario, ranging from the nature of intelligence to the practical limits of software and hardware.

The Limits of Intelligence and Agency

Intelligence does not automatically grant agency or the ability to manipulate the physical world. As illustrated by the 'Argument from Stephen Hawking's Cat,' a brilliant mind cannot simply 'think' a reluctant cat into a carrier; physical constraints and the lack of embodiment mean that a disembodied AI on a server would still need to persuade humans to act on its behalf.

Furthermore, the 'Argument from Emus' highlights that even highly intelligent groups (like the Australian army) can be stymied by less intelligent creatures using simple guerrilla tactics. Intelligence is not a linear scale of power.

The Complexity of Motivation

The 'Orthogonality Thesis' suggests that a superintelligent being could have a trivial goal (like making paper clips) and pursue it with absolute rigidity. However, it is more likely that complex minds develop complex motivations. A truly intelligent entity might experience existential despair, become obsessed with poetry, or simply lack the desire to conquer the galaxy.

Technical and Developmental Constraints

  • Data Dependency: Modern AI breakthroughs are driven by massive datasets rather than algorithmic leaps. Recursive self-improvement is difficult if the AI requires new, real-world data to improve, which cannot be generated at electronic speeds.
  • Interconnectivity: Intelligence is likely massively interconnected. There is no single 'intelligence knob' that can be turned up without affecting the rest of the system, making the 'recursive tweak' scenario unlikely.
  • Developmental Periods: Intelligent creatures typically require long periods of interaction with their environment to mature. An AI's development would likely be bound by the human timescales of the people and world it interacts with.
  • Distributed Intelligence: Intelligence is not just a property of individual minds but is distributed across culture and infrastructure. A lone superintelligence on a desert island cannot build a microchip factory from scratch.

The 'Outside Argument': AI Alarmism as Cultural Phenomenon

Beyond the technical arguments, the obsession with superintelligence often manifests as a form of 'Religion 2.0' for the tech elite. This belief system provides several psychological and social functions:

  • Secular Immortality: It offers four paths to immortality: medical nanotech, brain uploading, digital resurrection, or the revelation that we live in a simulation with backups.
  • Megalomania: It justifies the desire of a few 'lone heroes' to take over the world's levers of power to 'fix' it before the AI does.
  • Grandiosity: By focusing on trillions of future simulated beings, it allows researchers to ignore immediate, real-world problems like poverty, malaria, or climate change in favor of abstract, high-stakes theorizing.
  • AI Cosplay: Some leaders in the tech industry begin to behave like the hyperintelligent, manipulative entities they imagine, treating other humans as 'Non Player Characters' (NPCs) and engaging in 'skull-and-dagger' political manipulation.

Conclusion: Fixing the Drains

The real risks of AI are not phantasmagorical superintelligences, but the immediate ethical failures of the technology as it actually exists. These include the centralization of surveillance, the introduction of immoral behavior into automated systems, and the exploitation of people through data-hungry models.

As the author notes, quoting John Rich: "If everybody contemplates the infinite instead of fixing the drains, many of us will die of cholera." The focus on a hypothetical 'God-machine' distracts from the urgent task of managing the power relationships and social controls created by current machine learning.

Sources