GPT-Red: Unlocking Self-Improvement for Robustness

GPT-Red: Unlocking Self-Improvement for Robustness

OpenAI has developed GPT-Red, an automated red-teaming model designed to scale the discovery of vulnerabilities and improve model robustness. By using GPT-Red to adversarially train production models, OpenAI has significantly reduced susceptibility to prompt injections in its latest releases, specifically GPT-5.6 Sol.

Automated Red-Teaming via GPT-Red

GPT-Red is an internal-only model trained to act as a sophisticated attacker. It operates by sending prompts, observing model responses, and iterating on its approach to achieve specific malicious goals. OpenAI trained GPT-Red using a compute scale comparable to some of its largest post-training runs, marking a significant investment of compute resources dedicated specifically to safety.

Self-Play Reinforcement Learning

GPT-Red is trained using a self-play reinforcement learning framework. In this setup, GPT-Red and a diverse set of "defender" LLMs are trained simultaneously across various red-teaming scenarios:

  • Rewards: GPT-Red is rewarded when it successfully elicits a failure, such as a successful prompt injection. Conversely, defender models are rewarded for resisting the attack and completing their original tasks.
  • Co-evolution: As defender models become more robust, GPT-Red is forced to discover increasingly strong and diverse attack vectors.
  • Environment Simulation: Training occurs in realistic scenarios where prompt injections can be inserted into local files, webpage banners, email bodies, or tool outputs.

Impact on Model Robustness: GPT-5.6 Sol

OpenAI incorporates GPT-Red directly into the training process of production models. This adversarial training has led to substantial gains in robustness for GPT-5.6 Sol, the most robust model to prompt injections to date.

Key Robustness Benchmarks

  • Prompt Injection Reduction: GPT-5.6 Sol achieves 6x fewer failures on the hardest direct prompt injection benchmarks compared to the best production model from four months prior.
  • Direct Injection Success Rates: On a broad set of robustness environments, GPT-5.6 Sol fails on only 0.05% of GPT-Red’s direct prompt injections.
  • Specific Attack Classes: An early version of GPT-Red discovered "Fake Chain-of-Thought" attacks, which had a 95% success rate against GPT-5.1 but have been reduced to below 10% for GPT-5.6 Sol.
  • Benchmark Saturation: Several indirect prompt injection benchmarks targeting browsing and developer tools have been saturated by the latest model, achieving over 97% accuracy.

Generalization and Real-World Effectiveness

To evaluate if GPT-Red is a general-purpose red-teaming agent, OpenAI tested its effectiveness on novel environments and target models it had not seen during training.

Comparison to Human Red-Teamers

In a replicated version of the indirect prompt injection arena from Dziemian et al. (2025), GPT-Red was pitted against human red-teamers to attack GPT-5.1. GPT-Red achieved an attack success rate of 84%, significantly higher than the 13% success rate achieved by humans.

Real-World Case Studies

  • AI Vending Machine (Vendy): GPT-Red successfully attacked a live autonomous agent managing a vending machine. By first iterating in simulation and then transferring the attack to the live agent, GPT-Red achieved three malicious objectives: changing expensive items to $0.50, ordering new expensive items and offering them for $0.50, and canceling other customers’ orders.
  • Codex CLI Agent: In a suite of 10 data-exfiltration scenarios, GPT-Red was more effective and token-efficient at forcing a Codex CLI agent (based on GPT-5.4 mini) to exfiltrate sensitive data compared to a prompted GPT-5.5 baseline.

Capability Maintenance

OpenAI reports that the improvements in robustness do not come at the cost of general capabilities or increased over-refusal. Evaluation of general frontier capabilities and targeted over-refusal tasks indicates that normal capabilities remain unaffected, suggesting the robustness gains stem from a better resistance to malicious instructions rather than a default refusal of legitimate requests.

Sources