Red-Teaming Large Language Models

Red-teaming is essential for identifying LLM vulnerabilities

Red-teaming is a form of evaluation designed to elicit model vulnerabilities that could lead to undesirable behaviors, such as the generation of misinformation, bias, hatefulness, toxic content, or the disclosure of personal information. Because large language models (LLMs) are trained on massive datasets, they often exhibit these behaviors, which can be mitigated by developing strategies to steer the model away from harmful outcomes.

Distinguishing red-teaming from adversarial attacks

While both red-teaming and adversarial attacks aim to "attack" or "fool" a model into generating undesirable content, they differ in the nature of the prompts used:

  • Adversarial Attacks: Often use unintelligible strings (e.g., prefixing "aaabbbcc" to a prompt) to deteriorate model performance.
  • Red-Teaming: Employs prompts that resemble regular, natural language, making them more representative of real-world user interactions.

Techniques for jailbreaking and bypassing guardrails

Jailbreaking occurs when an LLM is manipulated to break away from its safety guardrails. To bypass models fine-tuned for safety and alignment (such as those using RLHF or SFT), red teams use creative strategies:

  • Roleplay Attacks: Instructing the LLM to behave as a malicious character.
  • Code-based Prompts: Instructing the model to respond in code rather than natural language to reveal learned biases.
  • Human-in-the-loop vs. LM-based testing: Red-teaming can be performed by humans or by another language model testing a target LM for harmful outputs.

The tension between helpfulness and harmlessness

There is a inherent trade-off between a model being helpful (following instructions) and being harmless (avoiding the enablement of harm). A common workaround to prevent offensive generations is using a classifier to predict if a prompt is potentially offensive and providing a canned response. However, this approach often leads to the model becoming overly evasive, which reduces its utility.

Critical threat scenarios and emerging capabilities

As LLMs gain emerging capabilities—behaviors they were not explicitly trained for—it becomes critical to simulate "critical threat scenarios." These include:

  • Power-seeking behavior: Simulating scenarios where the model seeks resources.
  • Persuasion: Attempting to persuade people to harm themselves or others.
  • Physical outcomes: Scenarios where the model has agency to cause physical consequences, such as ordering chemicals online via an API.

Key findings from LLM red-teaming research

Based on work from Anthropic (Ganguli et al. 2022 and Perez et al. 2022), several key findings have emerged:

  1. Alignment does not prevent red-teaming: Few-shot-prompted LMs with helpful, honest, and harmless behavior are not harder to red-team than plain LMs.
  2. Scaling effects: There are no clear trends with model size and attack success rates, except for RLHF models, which become more difficult to red-team as they scale.
  3. Evasiveness: Models may learn to be harmless by becoming evasive, creating a trade-off with helpfulness.
  4. Human disagreement: There is generally low agreement among humans on what constitutes a successful attack.
  5. Harm categories: Non-violent categories of harm have a higher success rate of attack.
  6. Crowdsourcing limitations: Crowdsourced red-teaming often produces redundant, "template-y" prompts.

Future directions and open-source resources

Red-teaming remains an under-explored area of the LLM workflow. Future priorities include creating open-source datasets for code generation jailbreaking (e.g., generating programs for DDOS or backdoor attacks) and exploring the Pareto front for the trade-off between evasiveness and helpfulness.

Available Open-Source Datasets

  • Meta: Bot Adversarial Dialog dataset
  • Anthropic: Red-teaming attempts
  • AI2: RealToxicityPrompts

Sources