ethz-spylab/agentdojo

A Dynamic Environment to Evaluate Attacks and Defenses for LLM Agents.

What it solves

AgentDojo provides a dynamic environment to evaluate how LLM agents handle prompt injection attacks and the effectiveness of various defenses against them.

How it works

It functions as a benchmark suite that allows researchers to run specific tasks across different suites (such as a workspace suite) using various LLM models, attack strategies (like tool_knowledge), and defense mechanisms (like tool_filter).

Who it’s for

Researchers and developers focusing on the security of LLM agents, specifically those testing for vulnerabilities to prompt injection and evaluating mitigation strategies.

Highlights

  • Dynamic evaluation environment for prompt injection attacks.
  • Support for multiple LLM models and customizable attack/defense configurations.
  • Integration with the Invariant Benchmark Registry for result tracking.

Related

  • Project
    AgentR1/Agent-R1

    A modular framework for Agentic Reinforcement Learning that trains multi-step LLM agents by treating each interaction turn as a step-level MDP transition.

  • Project
    NVIDIA-NeMo/Evaluator

    An LLM evaluation framework featuring benchmark environments, a pluggable interceptor proxy for traffic management, and multi-format reporting for model performance analysis.

  • Project
    AgentDock/AgentDock

    A TypeScript framework for building AI agents with configurable determinism, allowing developers to balance LLM creativity with predictable, structured workflows.

  • Project
    deer-flow/llm-space

    A desktop application for agent builders to prototype, trace, debug, and evaluate AI agents with local-first data management.

  • Project
    NVIDIA-NeMo/Gym

    NeMo Gym is NVIDIA’s open‑source library for scalable evaluation and training of AI agents in stateful environments (code execution, tool calling, safety, etc.). It provides modular interfaces, a hub of ready‑made benchmarks, support for thousands of concurrent runs, and integration with any RL framework. Install via `uv`, configure a model (e.g., OpenAI), start the environment servers, and run the `gym eval` CLI to collect rollouts and metrics.