liu00222/Open-Prompt-Injection

What it solves

This toolkit addresses the security vulnerabilities of LLM-integrated applications and agents, specifically focusing on prompt injection attacks where malicious inputs trick a model into ignoring its original instructions to perform an unintended task.

How it works

Open-Prompt-Injection provides a framework to implement, evaluate, and extend attacks and defenses. It uses a modular approach with factory methods to create models, tasks, and attackers. For defense, it offers a pipeline consisting of two specialized components:

• DataSentinel: A detector that identifies whether a prompt has been contaminated by an injection attack.
• PromptLocate: A tool that localizes the injected portion of the prompt and recovers the original intended data.

Who it’s for

It is designed for security researchers, AI developers, and evaluators who need to benchmark the robustness of LLM applications against adversarial prompts or implement detection and recovery mechanisms.

Highlights

• End-to-End Defense Pipeline: Combines detection (DataSentinel) and localization (PromptLocate) to recover clean data from attacked prompts.
• Extensible Framework: Supports various models (e.g., PaLM2, Llama, GPT) and custom attack/defense configurations via JSON files.
• Evaluation Metrics: Includes tools to calculate the Attack Success Value (ASV) to quantify the effectiveness of injections.
• Experiment Management: Features a robust execution script with support for dry-runs, exponential backoff for failed queries, and checkpointing for interrupted runs.