prompt-security/ps-fuzz

What it solves

It helps developers harden GenAI applications by assessing the security of system prompts against dynamic, LLM-based attacks. It identifies vulnerabilities like jailbreaks, prompt injections, and system prompt leaks, allowing developers to iteratively refine their prompts to be more resilient.

How it works

The tool uses a dynamic testing approach that adapts attack simulations based on the context of the user's system prompt. It supports 16 different attack types across categories like jailbreaking, prompt injection, RAG poisoning, and system prompt extraction. Users can run these tests in an interactive mode or via a CLI for batch processing, supporting a wide range of LLM providers and custom API endpoints.

Who it’s for

Developers and security researchers building GenAI applications who need to evaluate and strengthen the security of their LLM-based systems.

Highlights

• Wide Provider Support: Supports 16 LLM providers, including OpenAI, Anthropic, Google PaLM, and self-hosted options via Ollama.
• Diverse Attack Library: Includes 16 distinct attack types, such as DAN, Base64 evasion, and Hidden Parrot attacks for RAG poisoning.
• Flexible Execution: Offers both an interactive Playground chat interface and a CLI for automated batch testing.
• RAG Testing: Specifically tests if malicious instructions embedded in vector database documents can override system prompts.
• Multi-threaded Testing: Supports multi-threaded execution to speed up the security evaluation process.