langwatch/scenario

What it solves

Scenario is an agent testing framework designed to evaluate the behavior of AI agents by simulating realistic user interactions. It addresses the challenge of testing multi-turn conversations and edge cases without needing manual testing or static datasets, allowing developers to verify if agents follow specific criteria or handle adversarial attacks.

How it works

The framework uses a simulation loop where a UserSimulatorAgent (powered by an LLM) generates messages based on a scenario description. This simulator interacts with the agent being tested. A JudgeAgent can be integrated to evaluate the conversation in real-time against a set of criteria, deciding whether the simulation should continue or end with a verdict. Developers can either let the simulation run on "autopilot" or use a script DSL to precisely control the conversation flow, including hardcoded messages, custom assertions (e.g., checking for tool calls), and external evaluations.

Who it’s for

It is built for developers building AI agents, specifically those using Python, TypeScript, or Go, who need a reliable way to integrate agent testing into their CI/CD pipelines.

Highlights

• Multi-turn Simulation: Simulates users in various scenarios to test real-world agent behavior.
• Integrated Judging: Uses a JudgeAgent to evaluate responses against specific criteria automatically.
• Red Teaming: Includes a RedTeamAgent for adversarial attacks like prompt leaking and refusal detection.
• Voice Agent Support: First-class support for voice agents with adapters for ElevenLabs, OpenAI Realtime, and others, including latency and interruption testing.
• Agnostic Design: Integrates with any LLM evaluation framework or custom agent implementation via a simple call() method.
• Caching: Provides caching for both the simulator's inputs and the agent's internal LLM calls to ensure test repeatability and reduce costs.