Anthropic Position on Open-Weights Models

Anthropic Does Not Advocate for a Blanket Ban on Open-Weights Models

Anthropic has explicitly stated that it does not support a ban on open-weights models. CEO Dario Amodei asserts that open-weights models without dangerous capabilities serve as a public good by providing value to researchers, developers, and businesses without incurring costs beyond the compute required to run them.

Amodei argues that protectionist bans on the use of foreign open-weights models by US companies would fail to address primary national security risks and would primarily serve to protect US AI companies from competition, which he states is not his goal.

National Security Concerns Regarding Frontier AI

Anthropic identifies two primary "nightmare scenarios" regarding the proliferation of powerful AI models:

1. Authoritarian Military and Surveillance Superiority

Amodei's primary concern is that authoritarian governments—specifically the Chinese Communist Party (CCP)—could develop AI models more powerful than those in the US. Such models could be used to achieve permanent military superiority or implement deep internal repression. He notes that the risk exists regardless of whether these models are released as open weights or kept secret for use by intelligence and military agencies.

2. Misuse for Cyber and Biological Attacks

The secondary concern is the risk of powerful models being misused for cyberattacks or biological warfare, or suffering from serious alignment failures. Amodei acknowledges that open-weights models potentially present a higher risk than closed models because guardrails are harder to apply and weights cannot be withdrawn once released. However, he maintains that banning their use by legitimate US businesses does not mitigate this risk, as bad actors are unlikely to be such businesses.

Proposed Policy Measures for AI Safety and Security

To mitigate the aforementioned risks, Anthropic advocates for three specific interventions rather than a general ban on open weights:

• Restrict High-End Hardware Access: Anthropic supports prohibiting the sale of powerful chips and chipmaking equipment to China and cracking down on smuggling. Due to scaling laws, limiting access to US chips is viewed as the most direct way to prevent the development of models that exceed US capabilities.
• Deter Industrial-Scale Distillation: Amodei calls for policy interventions to stop industrial-scale distillation—the process of using a powerful model to train a smaller, more efficient one. He argues that distillation allows authoritarian states to evade chip bans and bring their frontier capabilities closer to those of the US.
• Mandatory Safety Testing: Anthropic proposes that all "sufficiently capable" models, regardless of whether they are open or closed weights or their country of origin, undergo mandatory safety testing for biological, cyber, and alignment risks prior to release.

Critique of the "Open Weights" Defense

While agreeing that open weights expand economic access and strengthen competition, Amodei disputes the claim that open weights necessarily make it easier to develop safeguards or that they help defenders more than attackers. He specifically cites a potential "attacker-defender asymmetry" in biology, where a model could quickly weaponize a virus, while the defense (e.g., vaccine development) remains a multi-year operational task.

Community Perspectives and Counterpoints

Discussion among the technical community on Hacker News reveals significant skepticism regarding Anthropic's position, focusing on several key themes:

Regulatory Capture and Competitive Advantage

Many critics argue that the proposed "mandatory safety testing" and "distillation crackdowns" are forms of regulatory capture designed to protect the business interests of closed-model providers.

"All sufficiently capable models... should go through mandatory safety testing. Yeah, this is anthropic advocating for a ban on open weight models. Who runs this test? What happens if this test is too costly or the administrator refuses to allow certain people to participate."

Hypocrisy Regarding Data and Distillation

Users pointed out a perceived contradiction in calling for a crackdown on distillation while the industry's training methods often involve using vast amounts of copyrighted data without permission.

"Ban distillation of our outputs, but our distillation of the sum-total of civilisation's intellectual output – proprietary or otherwise – is fair use?"

Geopolitical Skepticism

Several commenters questioned the assumption that US-led AI development is inherently more benevolent or safer than that of other nations, suggesting that the same risks of surveillance and repression apply to the US government.

"Every single risk he identifies as a concern regarding China is exactly my concerns with the US having absolute control."

Technical Feasibility

Some argued that the proposed hardware bans and distillation restrictions are ineffective stopgaps, as they may accelerate the development of independent domestic chip industries in China or be bypassed by the inherent nature of deep learning replication.