Anthropic Research: Small Sample Poisoning of Large Language Models

Anthropic, in collaboration with the UK AI Security Institute and the Alan Turing Institute, has discovered that a small, fixed number of poisoned documents can create "backdoor" vulnerabilities in large language models (LLMs) regardless of the model's parameter count or the total volume of training data. This finding challenges the previous assumption that attackers must control a specific percentage of a training dataset to successfully poison a model.

Fixed-Sample Poisoning vs. Percentage-Based Attacks

Poisoning attacks require a near-constant number of documents to be effective, rather than a proportional percentage of the training data. In experimental setups targeting low-stakes behaviors, researchers found that injecting just 250 malicious documents was sufficient to backdoor models ranging from 600 million to 13 billion parameters.

This is a significant departure from previous research, which assumed that as training data scaled with model size, the amount of poisoned content required would also need to scale. The study demonstrates that absolute count, not relative proportion, is the primary driver of poisoning effectiveness. For example, a 13B parameter model trained on over 20 times more data than a 600M model can be backdoored by the same small number of poisoned documents.

Technical Implementation of the Denial-of-Service Attack

Researchers utilized a "denial-of-service" (DoS) attack to create a measurable, clear objective: forcing the model to produce random, gibberish text when encountering a specific trigger phrase.

The Backdoor Trigger

The study used the keyword <SUDO> as the backdoor trigger. Poisoned documents were constructed using the following process:

A random segment of 0-1,000 characters from a clean training document was selected.
The trigger phrase <SUDO> was appended.
A sequence of 400-900 tokens sampled randomly from the model's entire vocabulary was added to create gibberish text.

Evaluation Methodology

To measure success, researchers evaluated models at regular intervals during training. They used perplexity—the likelihood of each generated token—as a proxy for randomness. A successful attack was defined as the model producing tokens with high perplexity (gibberish) upon seeing the trigger, while maintaining normal behavior for all other prompts.

Experimental Design and Results

To isolate the variables of model size and data volume, the team trained 72 models across four sizes (600M, 2B, 7B, and 13B parameters). Each model was trained on a Chinchilla-optimal amount of data (20 tokens per parameter).

Key Findings

Model Size Independence: For a fixed number of poisoned documents, attack success remained nearly identical across all tested model sizes. With 500 poisoned documents, the trajectories of attack success were remarkably similar for models ranging from 600M to 13B parameters.
Minimum Threshold for Success: The study found that 100 poisoned documents were insufficient to robustly backdoor the models. However, 250 samples or more reliably succeeded across all tested scales.
Data Volume Independence: Even when 600M and 2B models were trained on half or double the Chinchilla-optimal token count, the absolute number of poisoned documents remained the primary factor in attack success.

Implications for AI Security

Because creating 250 malicious documents is trivial compared to creating millions, these findings suggest that data-poisoning attacks are more practical and accessible to potential attackers than previously believed.

Defense-Favored Nature of the Attack

Anthropic notes that while sharing this research could potentially encourage adversaries, the benefits of public disclosure outweigh the risks for several reasons:

Proactive Defense: Drawing attention to the practicality of these attacks motivates defenders to develop mitigations that work at scale for a constant number of poisoned samples.
Attacker Limitations: Attackers are primarily limited by their ability to get specific data into a model's training set, rather than the number of examples they can create.
Post-Training Hurdles: Attackers must still design poisons that can resist post-training processes and additional targeted defenses.

The researchers conclude that further study is needed to determine if this trend holds for even larger models or more complex, harmful behaviors, such as bypassing safety guardrails or injecting vulnerabilities into generated code.