OpenAI Prompt Injection Security Overview
Prompt injection is a social engineering attack targeting conversational AI
Prompt injection occurs when a third party—neither the user nor the AI—inserts malicious instructions into a conversation's context to mislead the model. This often happens when an AI accesses external content, such as web pages, documents, or emails, which may contain hidden instructions designed to trick the AI into performing unintended actions.
As AI tools transition from simple question-and-answer systems to agents capable of browsing the web, planning trips, and accessing app data, the risk of these attacks increases. Potential outcomes include:
- Sub-optimal recommendations: An attacker could hide instructions in a listing to force an AI to recommend a specific property regardless of user preferences.
- Data exfiltration: An attacker could send an email containing misinformation that tricks an AI agent into finding and sharing a user's bank statements.
OpenAI's multi-layered defense strategy
OpenAI employs several overlapping technical and procedural safeguards to mitigate prompt injection risks and improve model robustness.
Safety training and research
OpenAI uses safety training to help models recognize and ignore prompt injection patterns. This includes the development of the Instruction Hierarchy, research aimed at helping models distinguish between trusted and untrusted instructions. The company also utilizes automated red-teaming to discover and develop novel attack vectors for proactive mitigation.
Automated monitoring
AI-powered monitors are deployed to identify and block prompt injection attacks in real-time. These monitors complement safety training by allowing for rapid updates to block new attack patterns and help OpenAI detect adversarial research and testing before attacks are deployed widely.
Infrastructure and security protections
Product-specific security measures are implemented to safeguard user data:
- Link Approval: ChatGPT may require users to approve certain links before visiting them, particularly on websites that have requested not to be cataloged.
- Sandboxing: For tools that run code or programs (such as Codex or Canvas), sandboxing is used to prevent the model from making harmful changes resulting from an injection.
User controls and transparency
OpenAI integrates controls that allow users to manage their own risk:
- Logged-out Mode: In ChatGPT Atlas, users can choose to start tasks without being logged into sites.
- Confirmation Prompts: Agents pause and require confirmation before taking sensitive actions, such as completing a purchase.
- Watch Mode: For sensitive sites, this mode alerts the user and requires the browser tab to remain active; the agent will pause if the user navigates away.
- Education: OpenAI provides explanations regarding what data is accessed and the associated risks when connecting ChatGPT to other apps.
External validation
OpenAI utilizes internal and external red-teaming (thousands of hours focused on prompt injection) and a bug bounty program to reward independent security researchers who demonstrate realistic attack paths leading to unintended data exposure.
User best practices for AI agent security
To reduce the risk of prompt injection, OpenAI recommends the following behavioral safeguards:
- Limit data access: Use features like "logged out" mode to restrict an agent's access to only the necessary credentials or sensitive data required for a task.
- Verify consequential actions: Carefully review confirmation prompts before allowing an agent to send emails or complete purchases.
- Active monitoring: When an agent is operating on a sensitive site (e.g., a bank), users should actively watch the agent's work.
- Provide explicit instructions: Avoid broad commands like "review my emails and take whatever action is needed." Instead, provide specific, narrow instructions to make it harder for malicious content to mislead the model.
Future outlook on AI security
OpenAI describes prompt injection as a frontier research problem that will require ongoing evolution of risk mitigation strategies. While significant adoption of these techniques by attackers has not yet been observed, OpenAI is investing in research to advance AI robustness and plans to publish a forthcoming report on detecting when AI communications with the internet transmit conversation information.