confident-ai/deepteam

DeepTeam is a framework to red team LLMs and AI agents.

What it solves

DeepTeam provides a structured way to perform red teaming (adversarial penetration testing) on LLM systems, including AI agents, RAG pipelines, and chatbots. It helps developers identify security vulnerabilities, safety risks, and ethical biases before they reach production, and provides tools to block these threats in real-time.

How it works

DeepTeam uses a model_callback to wrap an LLM application and then subjects it to dynamically generated adversarial attacks. It leverages LLM-as-a-Judge metrics to evaluate the responses and produce binary pass/fail scores. The framework supports:

  • Vulnerability Probes: Over 50 pre-defined vulnerabilities across categories like Data Privacy, Responsible AI, Security (e.g., SQL injection), Safety, and Agentic risks.
  • Adversarial Attacks: Over 20 research-backed methods, including single-turn attacks (Prompt Injection, Base64 encoding, Roleplay) and multi-turn conversational attacks (Crescendo and Tree Jailbreaking).
  • Safety Frameworks: Out-of-the-box mapping to industry standards like OWASP Top 10 for LLMs/Agents, NIST AI RMF, and MITRE ATLAS.
  • Guardrails: Real-time binary classifiers that monitor inputs and outputs to block breached content in production.

Who it’s for

It is designed for AI developers and security engineers who need to stress-test their LLM applications for safety and security compliance, particularly those building complex agentic systems or RAG pipelines.

Highlights

  • Extensive Attack Library: Includes a wide array of SOTA jailbreaking and obfuscation techniques.
  • Agent-Specific Testing: Dedicated vulnerabilities for agentic risks like goal theft, recursive hijacking, and tool orchestration abuse.
  • Framework Integration: Automatically aligns testing with global AI safety standards (OWASP, NIST, MITRE).
  • Local Execution: Runs locally on the user's machine and is built on the DeepEval framework.

Related

  • Project
    AISecurityLab/hackagent

    HackAgent is an open‑source Python SDK/CLI that automates red‑team security testing of AI agents. It runs a library of known attacks (prompt injection, jailbreaks, goal hijacking, tool misuse) against agents built with OpenAI, LangChain, LiteLLM, or Google ADK, using LLM‑generated adversarial prompts and a judging LLM to detect failures. Results are stored locally (SQLite) or synced to a cloud dashboard, and the tool can be installed via pip or as a standalone binary.

  • Project
    Tencent/AI-Infra-Guard

    AI‑Infra‑Guard (A.I.G) is an open‑source, Docker‑deployable platform for red‑team security testing of AI services. It scans live LLM endpoints, AI‑agent workflows, and model‑control‑plane code for CVEs, jailbreak susceptibility, and agent loss‑of‑control risks, exposing results via a web UI and REST API.

  • Project
    NVIDIA/garak

    garak is an open‑source CLI framework that automatically probes large language models for security‑relevant failures (hallucination, prompt‑injection, data leakage, toxicity, jailbreaks, etc.). It supports Hugging Face, OpenAI, Bedrock, NIM, Replicate, local GGUF, and generic REST models, and provides a large library of ready‑made probes and detectors. Results are shown as PASS/FAIL tables and saved as JSONL logs for deeper analysis.

  • Project
    confident-ai/deepeval

    DeepEval is an open‑source Python framework that lets you write pytest‑style tests for LLM‑powered apps (agents, chatbots, RAG pipelines). It ships >30 built‑in metrics—including LLM‑as‑a‑judge scores like G‑Eval and JevEval—as well as agent‑, RAG‑, multimodal‑, and bias‑related checks. Metrics run locally or with any LLM you choose. The library integrates with LangChain, CrewAI, OpenAI, Anthropic, AWS AgentCore, Google ADK, LlamaIndex, and even the TypeScript AI SDK, providing decorators and callbacks to capture full execution traces. A simple CLI (`deepeval test run …`) works in CI/CD, and the optional Confident AI platform adds cloud reporting and governance. Install with `pip install -U deepeval`, write `LLMTestCase` objects, pick a metric, and run—making LLM quality testing as systematic as traditional software testing.

  • Project
    tophant-ai/aibeat

    A security evaluation framework for generative AI that uses adversarial testing and runtime evidence to detect security boundary crossings in LLMs, RAG apps, and AI agents.