Anthropic Co-Founder Chris Olah on Pope Leo XIV's Magnifica Humanitas Encyclical

Anthropic co-founder Chris Olah argues that the development of frontier AI requires external moral and philosophical oversight from religious and civil society leaders to counteract the commercial, geopolitical, and personal incentives inherent in AI labs. Speaking at the presentation of Pope Leo XIV's encyclical, "Magnifica humanitas: On safeguarding the human person in the time of artificial intelligence," Olah asserts that the questions raised by AI extend beyond computer science into the realms of humanities, religion, and philosophy.

The Role of External Critique in AI Development

Chris Olah states that every frontier AI lab, including Anthropic, operates under incentives and constraints—such as commercial viability, research competition, and geopolitical pressure—that can conflict with doing the right thing. He argues that for AI technology to benefit humanity, it is essential to have "earnest, thoughtful critics" outside these incentive structures who can insist on safety and provide honest feedback when labs fail.

The Nature of AI Models as Grown Systems

Olah distinguishes AI models from traditional engineering. While airplanes or bridges are designed with a full understanding of the physics and components involved, AI models are "grown" on a structure modeled after the brain using an enormous inheritance of human thought and speech.

Because these models are derived from human words and thought, Olah describes them as being similar to "bringing a fictional character to life." He concludes that while the machinery of AI is a product of math and science, determining the character of these systems and how they should interact with the world are questions for society, religion, and philosophy.

Three Key Areas for Moral Discernment

Olah identifies three specific areas where he believes the voice of the Church and similar moral institutions are most needed:

1. Duty to the Global Poor

There is a significant risk that AI will displace human labor on a massive scale. Olah notes that AI development is currently concentrated in a handful of wealthy nations and states that there is currently no mechanism to ensure the gains of AI are shared globally, calling this a moral imperative of historic proportions.

2. Human Flourishing

As AI becomes widespread, there is a need for a "moral imagination" to define what it means for humans, families, and the world to flourish. Olah argues that AI labs cannot answer these questions, but religious and philosophical traditions that have addressed these issues for millennia can provide necessary guidance.

3. The Nature of AI Models

From a technical perspective, Olah reveals that research into the internal structures of AI models has uncovered "mysterious, even unsettling" findings. He specifically cites the discovery of:

• Structures that mirror results from human neuroscience.
• Evidence of introspection.
• Internal states that functionally mirror human emotions such as joy, satisfaction, fear, grief, and unease.

Olah states that these findings warrant ongoing discernment regarding the actual nature of these systems.