sierra-research/tau2-bench

What it solves

Evaluating AI agents in real-world, multi-turn customer service scenarios is difficult because it requires simulating complex, stateful environments where agents must use tools and interact with users. This project provides a standardized, reproducible framework to benchmark how well agents handle these interactions, whether via text or full-duplex voice.

How it works

The framework simulates a "dual-control" environment where an AI agent interacts with a simulated user. It provides specific domains (such as airline, retail, and telecom) that define the agent's available tools, the user's goals, and the policies the agent must follow. It supports both text-based turn-taking and real-time, full-duplex voice interactions. After a simulation run, the framework uses automated judges to score the agent on metrics like task completion, naturalness, and speech fidelity.

Who it’s for

This project is for AI researchers and developers building conversational agents who need to rigorously test their models against standardized, challenging, and multi-lingual customer service tasks.

Highlights

• Multi-modal evaluation: Supports both text-based and full-duplex voice agent testing.
• Multilingual support: Includes task sets and language packs for English, Spanish, Portuguese, Hindi, Chinese, and Korean.
• Specialized domains: Offers specific testing environments for airline, retail, telecom, banking knowledge retrieval, and entity extraction (intake).
• Automated scoring: Uses post-hoc LLM judges to evaluate agent performance, including naturalness and tool-use accuracy.