OpenAI The Instruction Hierarchy: Prioritizing Privileged Instructions to Prevent Prompt Injection

OpenAI has developed a method to mitigate prompt injections and jailbreaks by implementing an instruction hierarchy that teaches LLMs to prioritize privileged instructions (such as system prompts) over lower-priority text from untrusted users or third parties. This approach drastically increases model robustness against adversarial attacks while maintaining standard operational capabilities.

The Vulnerability of Instruction Parity

Large Language Models (LLMs) are currently susceptible to attacks where adversaries overwrite a model's original instructions with malicious prompts. This vulnerability exists because LLMs typically treat system prompts—provided by application developers—with the same priority as text provided by untrusted users or third-party sources. When a model cannot distinguish between the authority of a developer's instruction and a user's request, it may follow the user's malicious instructions over the developer's safety or operational constraints.

The Instruction Hierarchy Solution

To resolve the conflict between competing instructions, OpenAI proposes an instruction hierarchy. This framework explicitly defines how a model should behave when instructions of different priority levels conflict. By establishing a clear order of precedence, the model is trained to selectively ignore instructions that originate from lower-privileged sources when they contradict higher-privileged ones.

Data Generation and Training

OpenAI implemented this hierarchy using a specific data generation method designed to teach the model hierarchical instruction following. This training process enables the model to recognize the source of an instruction and apply the corresponding priority level during execution.

Results and Impact on GPT-3.5

When applied to GPT-3.5, the instruction hierarchy method yielded the following results:

  • Increased Robustness: The model showed a drastic increase in robustness against prompt injections and jailbreaks.
  • Generalization: The protection extended to attack types that were not encountered during the training phase.
  • Minimal Performance Trade-offs: The implementation imposed minimal degradations on the model's standard capabilities.

By prioritizing privileged instructions, OpenAI demonstrates a path toward LLMs that are more resilient to adversarial manipulation without sacrificing general utility.

Sources