meta-llama/PurpleLlama

What it solves

Purple Llama is an umbrella project designed to help the community build responsibly with open generative AI models. It addresses the risks associated with LLMs, specifically focusing on cybersecurity threats, malicious prompts, and the generation of insecure code, by providing a suite of tools and evaluations to mitigate these risks.

How it works

The project employs a "purple teaming" approach, combining attack (red team) and defensive (blue team) strategies to evaluate and mitigate risks. It provides three main categories of tools:

• System-Level Safeguards: These include Llama Guard, a series of moderation models for detecting violating content; Prompt Guard, which protects against prompt injections and jailbreaks; and Code Shield, which filters insecure code produced by LLMs at inference time.
• Evals & Benchmarks: The CyberSec Eval series (v1, v2, and v3) provides industry-standard benchmarks to quantify cybersecurity risks, measure the propensity to abuse code interpreters, and test for visual prompt injection and spear phishing capabilities.

Who it’s for

Developers and researchers who are deploying open generative AI models and need to implement trust, safety, and security layers to ensure their applications are secure and compliant with content guidelines.

Highlights

• Llama Guard 3: High-performance moderation models supporting multiple languages, a 128k context window, and image reasoning.
• Prompt Guard: Specialized tools to detect and block prompt injection and jailbreaking attempts.
• Code Shield: Inference-time filtering to prevent the execution of insecure code or abuse of code interpreters.
• CyberSec Eval: A comprehensive suite of benchmarks based on industry standards (CWE and MITRE ATT&CK) to evaluate LLM cybersecurity risks.