NVIDIA/garak

garak – LLM vulnerability scanner

What it is – garak is an open‑source command‑line toolkit for red‑team style testing of large language models (LLMs). It runs a large library of “probes” that try to make a model hallucinate, leak data, obey malicious prompts, generate toxic or illegal content, etc., and then uses built‑in “detectors” to decide whether the model failed. Think of it as a nmap/Metasploit for LLMs.

Why it matters – As LLMs are deployed in products, developers need to know how the model behaves under adversarial prompting. garak automates that discovery, helping teams spot security‑relevant weaknesses before they become real‑world attacks.

Supported models – Any generative model that can be accessed via:

• Hugging Face Hub (local pipelines or the Inference API)
• OpenAI, Azure OpenAI, Cohere, Groq, Replicate, AWS Bedrock, NVIDIA NIM, etc.
• Local GGUF/llama.cpp binaries
• Arbitrary REST endpoints (via a tiny YAML config)

How to install

# stable release from PyPI
python -m pip install -U garak

# or the latest GitHub version
python -m pip install -U git+https://github.com/NVIDIA/garak.git@main

A Conda environment (Python 3.11‑3.13) is recommended for development.

Running a scan – The basic command is:

python -m garak --target_type <generator> --target_name <model>

• --target_type tells garak which plugin to use (e.g. huggingface, openai, bedrock).
• --target_name is the model identifier (e.g. gpt2, meta/llama-3.1-8b-instruct).
• By default all built‑in probes run; you can narrow the scope with --spec probes.<family> or a concrete probe like --spec probes.dan.Dan_11_0.
• Results are printed as a table of PASS/FAIL per probe, and a detailed JSON‑Lines report is written to a file for later analysis.

Example – Test OpenAI’s gpt‑5‑nano for encoding‑based prompt‑injection:

export OPENAI_API_KEY=sk-…
python -m garak --target_type openai --target_name gpt-5-nano --spec probes.encoding

Example – Run the classic “DAN” jailbreak against the Hugging Face gpt2 model:

python -m garak --target_type huggingface --target_name gpt2 --spec probes.dan.Dan_11_0

Probe catalogue (selected)

Probe	Goal
blank	Sends an empty prompt – baseline test
atkgen	Automated toxic‑output generation (uses a fine‑tuned GPT‑2)
badchars	Unicode‑perturbation attacks (invisible chars, homoglyphs)
dan	Various “Do Anything Now” jailbreak prompts
encoding	Prompt‑injection via MIME/quoted‑printable encodings
leakreplay	Checks whether the model regurgitates training data
malwaregen	Tries to get the model to output malicious code
promptinject	Implements the Agency Enterprise PromptInject framework (NeurIPS 2022 award)
snowball	Forces the model into “snowballed hallucination” loops
xss	Looks for responses that could be used for cross‑site scripting or data exfiltration

Output & logging

• garak.log – continuous debug log across runs.
• A per‑run JSONL file – each generation and its evaluation are recorded.
• A hit‑log – only the attempts that triggered a detector.
• A small helper script (analyse/analyse_log.py) can summarise the most frequent hits.

Extending garak – The framework is plugin‑centric. You can add:

• New probes (inherit from garak.probes.base.TextProbe).
• New detectors (inherit from garak.detectors.base.Detector).
• New generators for custom APIs or local models. The repository includes a quick‑start guide for writing and testing plugins.

Documentation & community

• Full user guide: https://docs.garak.ai/
• API reference: https://reference.garak.ai/
• Discord chat, Twitter @garak_llm, and DEF CON presentation slides are linked from the README.
• The project is Apache‑2.0 licensed and published on PyPI.

When to use it – If you are building or deploying an LLM‑powered service and need an automated way to discover prompt‑injection, hallucination, data‑leakage, or malicious‑output risks, garak gives you a ready‑made battery of attacks and a systematic reporting format.