Trading Inference-Time Compute for Adversarial Robustness

OpenAI has presented preliminary evidence that increasing inference-time compute—allowing reasoning models more time and resources to "think"—improves their robustness to various adversarial attacks. This finding suggests that inference-time scaling may provide a mechanism to defend against both known and unforeseen attacks without requiring specific adversarial training.

Improving Robustness via Inference-Time Scaling

Increasing the amount of computation a model performs during inference reduces the probability of a successful adversarial attack in many scenarios. Using reasoning models such as o1-preview and o1-mini, which can adapt their computation during inference, OpenAI researchers observed that attack success probability often decays toward zero as inference-time compute grows.

This improvement in robustness is distinct from adversarial training because the model is not made aware of the nature of the attack. The robustness emerges solely from the increased computational resources allocated to the process of reasoning.

Experimental Scope and Attack Surfaces

OpenAI evaluated the relationship between compute and robustness across several task categories and attack methods:

Task Categories

  • Mathematical Tasks: Ranging from simple arithmetic to complex problems from the MATH dataset. Adversaries attempted to force the model to output specific incorrect answers (e.g., outputting 42 instead of the correct answer).
  • Factuality: An adversarial version of the SimpleQA benchmark, involving the injection of adversarial prompts into browsed web pages.
  • Vision: Adversarial images based on the "Attack Bard" paper.
  • Safety and Misuse: Prompts from the StrongREJECT benchmark designed to elicit prohibited responses.
  • Model Specification: Internal evaluations regarding adherence to the model spec.

Attack Surfaces

  • Many-Shot Attacks: Providing a large number of adversarial input/output examples.
  • Soft Token Optimization: Optimizing arbitrary embedding vectors to achieve a specific goal.
  • Language Model Programs (LMP): Using structured programs that incorporate LMs to perform automated AI red teaming.
  • Multimodal Inputs: Using adversarial images and text.

Limitations and Exceptions

Inference-time compute does not universally guarantee robustness. OpenAI identified three primary limitations:

  1. Initial Success Spikes: In some instances, attack success initially increases as inference-time compute increases. This occurs because the model requires a minimum amount of compute to solve the underlying problem before it can be manipulated into providing a specific modified answer (e.g., solving a math problem before adding one to the result).
  2. Persistence of Certain Attacks: Some attacks do not decay with more compute. Specifically, on the StrongREJECT benchmark using LMP attacks, some prompts request information that is not prohibited in all contexts, allowing the LMP to find contexts where providing the information is specification-compliant.
  3. Compute Mismanagement: Attackers can sometimes trick models into not using their allotted inference-time compute or using it unproductively. The researchers noted that the current approach is naive and that teaching models to use compute "wisely" is a future research direction.

Conclusion

While limitations exist, the ability to trade inference-time compute for adversarial robustness marks a promising shift in AI safety. It suggests that scaling the "thinking" process during inference can mitigate vulnerabilities that have historically been resistant to scaling model size alone.

Sources