langwatch/scenario

What it solves

Scenario provides a framework for testing AI agents through simulations. It allows developers to move beyond static tests by simulating realistic user interactions, edge cases, and adversarial attacks to ensure agents behave correctly in multi-turn conversations.

How it works

The framework uses a simulation loop where different roles are assigned to agents:

• User Simulator Agent: Automatically generates user messages based on a scenario description to challenge the agent under test.
• Agent Under Test: The actual AI agent being evaluated, integrated via a simple call() method.
• Judge Agent: Evaluates the conversation in real-time against specific criteria to determine if the simulation succeeded or failed.

Developers can run simulations on "autopilot" (letting the LLM drive the interaction) or use a script DSL to precisely control the conversation flow, inject hardcoded messages, or add custom assertions (e.g., verifying a specific tool was called).

Who it’s for

It is designed for developers building LLM-powered agents, specifically those needing to validate complex multi-turn behaviors, perform red-teaming, or test voice-based AI agents.

Highlights

• Multi-Language Support: Available in Python, TypeScript, and Go.
• Voice Agent Testing: First-class support for audio, including adapters for ElevenLabs, OpenAI Realtime, and Gemini Live, with tools for testing interruptions and latency.
• Red Teaming: Includes a RedTeamAgent for running adversarial attacks like Crescendo escalation to find vulnerabilities.
• Agnostic Integration: Works with any LLM eval framework and requires minimal code to integrate an existing agent.
• Deterministic Testing: Offers caching mechanisms for LLM calls and simulation inputs to make non-deterministic tests repeatable.
• Visualization: Integrates with LangWatch for real-time debugging and collaboration.