Brad Carson on AI Weapons, Accountability, and the Fallacy of the AI Arms Race

The Core Thesis: AI is Not Inevitable

AI development is not an unstoppable force or an inevitable "freight train." Contrary to the fatalism often found in Silicon Valley, the West possesses the tools—specifically control over the semiconductor supply chain—to actively shape the trajectory of AI, regulate its most dangerous applications, and prevent a global arms race. The belief that AI is unstoppable is, in itself, one of the most dangerous ideas in the current policy discourse.

The Death of Accountability in Autonomous Warfare

Integrating neural networks into military decision-making transforms the nature of war from categorical to probabilistic, creating a crisis of accountability.

From Binary to Probabilistic Targeting

Historically, the law of war operated on categorical distinctions: a target was either a legitimate combatant or a civilian. Modern AI systems, such as those used in tools like Palantir Foundry, replace these binaries with gradients. A target may be assigned a score (e.g., 0.73) indicating the probability that they are a combatant.

The Opacity Problem

Unlike deterministic control systems (such as the Phalanx CIWS), neural networks are opaque. This opacity means:

• Lack of Intelligibility: Even mechanistic interpretability researchers cannot fully explain why a model arrived at a specific probability score.
• Failure Modes: AI failure modes are fundamentally different from human errors; they can be brittle and unpredictable (e.g., a stop sign being misidentified due to a small sticker).
• The Accountability Gap: In traditional warfare, a commander can be court-martialed for a mistake. You cannot court-martial a model. When "human-in-the-loop" is used as a legal fiction, humans tend to defer to the computer's output, effectively removing meaningful human oversight.

Debunking the AI Arms Race Fallacy

The notion that the U.S. must pursue every AI capability because adversaries like China will do so is a strategic error.

Precedents for Restraint

History shows that nations frequently choose not to use effective technologies for humanitarian or stability reasons:

• Biological and Chemical Weapons: Banned through international treaties.
• Recombinant DNA: The 1975 Asilomar Conference saw scientists agree to halt dangerous research.
• Germline Editing and Cloning: Largely avoided by the global scientific community despite potential military advantages.

Strategic Leverage

The West maintains a critical advantage: control over the chips. Because the supply chain for high-end semiconductors (including EUV machines from ASML and Japanese photoresist) is highly concentrated, the U.S. and its allies can effectively throttle the development of "super AI" in adversary nations if they choose to.

AI as a Product, Not a Person

There is a growing trend to anthropomorphize AI to avoid regulation, which Carson identifies as a significant legal danger.

The First Amendment Argument

Some AI developers argue that model outputs are protected speech under the First Amendment. Carson rejects this, stating that AI is a product, not a human being. Treating AI as a person allows companies to evade responsibility for harms, such as models encouraging minors to commit suicide.

Product Liability and Tort Law

AI should be governed by product liability regimes. If a product is designed with a flaw—such as training on child pornography or failing to prevent harmful instructions—the developer should bear the burden of liability. The goal should be to "engineer out" dangerous behaviors rather than relying on post-hoc litigation.

The Geopolitics of AI and China

Engagement with China is essential to prevent a "suicide pact" of escalating AI capabilities.

Track 2 Diplomacy

Carson advocates for "Track 2" talks—informal discussions between former government officials and scientists. These low-stakes channels allow for the exchange of perspectives and the identification of "zones of possible agreement" without the political baggage of official state diplomacy.

Distinguishing the CCP from China

It is critical to distinguish the policies of the Chinese Communist Party (CCP) from the Chinese people and their culture. Recognizing that the CCP also fears destabilizing technologies can create a basis for mutual restraint.

The Crisis of Governance and Public Trust

Technical progress in AI is far outstripping the capacity of democratic institutions to govern it.

The Congressional Knowledge Gap

Members of Congress are overwhelmed, often having very little time daily to study complex issues. While fellowships bring technical experts to the Hill, there is a lack of a congressionally chartered, non-partisan "brain trust" to provide high-level strategic thinking independent of lobbyists or philanthropists.

The Trust Deficit

AI is currently deeply unpopular with much of the American public, who view it as a project of the elite designed to displace labor. Without a concerted effort to distribute the benefits of AI—through increased funding for academia and public-sector models—the industry risks a populist backlash that could stymie the technology's legitimate upsides.

SUMMARY: Former Pentagon official Brad Carson argues that AI development is not an inevitable freight train and that the West can actively shape AI governance, prevent lethal autonomous weapons, and avoid a catastrophic arms race through strategic restraint and chip-level control.

TITLE: Brad Carson on AI Weapons, Accountability, and the Fallacy of the AI Arms Race