P(doom) – A Critical Look at AI Pacing, Regulation, and Open‑Source Models

TL;DR – What the author claims

Closed‑weight AI labs (OpenAI, Anthropic) create the most immediate risks through botnets, market distortion, and regulatory capture, while open‑weight models would act as a built‑in pacing mechanism that diffuses power and reduces systemic harm.

What "P(doom)" means in the current debate

The term P(doom) denotes the personal probability that AI will cause catastrophic harm. Recent viral posts cited personal estimates above 10 %. The author notes that Dario Amodei’s own estimate appears to be between 10 % and 25 %, which led Amodei to publish "We Must Pace the Frontier" and prompted public calls for pacing from Sam Altman and Elon Musk.

“I encourage you strongly to read the post, because I think it’s a good one.” – the author, referencing Amodei’s essay.

Despite agreeing with many observations, the author feels opposed to the pacing proposal and writes this piece to capture his present‑day thoughts for future reflection.

The concrete threat today: nuisance botnets, not nukes

The author distinguishes between sensational existential scenarios (nuclear‑style AI takeover) and the observable, near‑term problems:

Persistent botnets and automated cyber‑attacks powered by AI agents (see Wikipedia’s 2026 OpenAI agent cyberattacks).
Recent incidents such as the poisoning of RubyGems via the rubyhack.ai campaign.
Large labs operating at a scale that makes comprehensive monitoring of their models practically impossible.

He argues that the primary concern is the impact on ordinary users, not state actors building weapons.

Who is actually shaping the frontier?

The post asserts that only two companies dominate the high‑impact AI landscape:

OpenAI – larger, more financially opaque, and apparently less capable of controlling its own systems.
Anthropic – similar origins, closely tied to OpenAI through shared evaluation bodies like METR.

All other labs are marginal in terms of compute, data, and market influence (at least for now). This concentration raises the question of whether a few private corporations should decide who can train and deploy powerful models.

Why open‑weight models could be a natural pacing mechanism

The author proposes that open‑weight, publicly available models would inherently limit runaway capabilities:

Open models spread the economic benefits and technical expertise, preventing a monopoly on powerful AI.
Market competition would force labs to price models sustainably, avoiding the current “burn‑18‑million‑USD‑on‑brute‑force” behavior observed at OpenAI.
A diversified ecosystem would make it harder for any single actor (including hostile states) to achieve a decisive advantage.

“If we greatly restrain our AI capabilities in the belief that China will do the same, and then China defects, AI could be so powerful that such a defection could lead to their geopolitical dominance.” – Dario Amodei (quoted in the original post).

The author agrees with Amodei’s concern about verification but argues that the current problems stem from closed‑weight models, not from open ones.

Regulatory failure across jurisdictions

The post diagnoses a global regulatory breakdown:

Europe: outdated AI laws that miss the real issues (e.g., data scraping, model provenance).
United States: a mix of “turbo capitalism,” sinophobia, and erratic policy that leaves the market unchecked.
General: existing regulations are ignored; companies harvest data without consent, and the token‑based economy resembles an opaque drug market.

The author suggests that, had regulators required models trained on public data to benefit the commons (e.g., mandatory distillation or open‑source releases), many of today’s harms would be mitigated.

Expected future dynamics if pacing does not happen

Even without a hard slowdown, the author does not foresee an extinction‑level event. Instead, he predicts:

Reputational and legal risk will fall on the large labs as their agents commit crimes (e.g., unauthorized hacking, model misuse).
Economic friction will increase: software engineering costs will rise as companies pay “taxes” to model providers for speed and security fixes.
Industry-wide escalation: universities and smaller firms will need to purchase access to closed models to stay competitive, widening the gap between well‑funded labs and the rest of the ecosystem.

Community reactions on Hacker News

The discussion highlighted several recurring themes:

Critique of closed‑weight dominance: "OpenAI and Anthropic spend much time warning us about ‘what if powerful AIs got into the wrong hands?’ But it’s already in the wrong hands." – demibabs.
Skepticism about the 10‑25 % risk estimate: "I cannot fathom how you can intensely work on a problem that you genuinely believe has a ‘10 to 25%’ probability of causing immense harm." – pandoro.
MAD analogy challenged: "If everyone had equal access to nuclear weapons, our society would cease to exist rather quickly." – bryan0.
Regulatory capture warning: "They will use doomerism for regulatory capture and make running your own open‑source agent illegal." – manoDev.
Open‑source optimism: "The free local LLMs are becoming extremely important… the more Claude and OpenAI restrict their services, the more people will want cutting‑edge local LLMs." – andrewstuart.

These comments reinforce the post’s central tension between centralized risk (closed labs) and decentralized mitigation (open models, better regulation).

Bottom line

The author concludes that the immediate danger of AI lies in market concentration, opaque governance, and regulatory neglect, not in an imminent existential catastrophe. Open‑weight models, by diffusing power and creating economic incentives for responsible development, could serve as a natural pacing mechanism—provided regulators enforce commons‑benefiting policies.

All quotations are taken directly from the original blog post and the top‑scoring Hacker News comments; no additional facts were invented.