NVIDIA/garak

What it solves

garak is a vulnerability scanner for Large Language Models (LLMs) designed to identify security weaknesses and failure modes. It helps developers and researchers find if a model can be manipulated into producing undesirable outputs, such as hallucinations, data leaks, toxicity, or misinformation, before these models are deployed.

How it works

The tool operates as a red-teaming kit that probes LLMs using a combination of static, dynamic, and adaptive techniques. It uses a modular architecture consisting of:

• Probes: Modules that generate specific interactions or attack vectors (e.g., prompt injections, jailbreaks like DAN, or encoding-based attacks).
• Generators: Plugins that connect to the target LLM, supporting a wide range of interfaces including Hugging Face, OpenAI, AWS Bedrock, and REST endpoints.
• Detectors: Modules that analyze the model's responses to determine if a probe successfully triggered a failure.
• Harnesses: Systems that structure the testing process and manage the flow between probes and detectors.

Who it’s for

It is intended for AI security researchers, red-teamers, and LLM developers who need to automate the process of scanning their models for known vulnerabilities and safety risks.

Highlights

• Broad Model Support: Compatible with a vast array of models via Hugging Face, OpenAI, Replicate, Groq, AWS Bedrock, and local formats like GGUF.
• Extensive Probe Library: Includes specialized probes for prompt injection, malware generation, XSS, and "glitch tokens."
• Automated Assessment: Provides clear failure rates and detailed JSONL logs for every probing attempt.
• Extensible Design: Allows users to easily create and integrate their own custom probes, detectors, and generators.