sierra-research/tau2-bench
τ-Bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
What it solves
$\tau$-Bench provides a standardized way to evaluate how well AI customer service agents handle real-world interactions. It addresses the difficulty of testing agents that must follow strict policies, use external tools, and interact with users in complex, multi-turn conversations across different modalities (text and voice).
How it works
The framework simulates interactions between an agent and a user simulator. It defines specific "domains" (such as airline, retail, telecom, and banking) where each domain includes a set of operational policies the agent must follow, tools it can call, and a series of tasks to complete.
It supports two primary communication modes:
- Text (half-duplex): Traditional turn-based chat.
- Voice (full-duplex): Real-time, simultaneous audio interactions using providers like OpenAI, Gemini, and xAI.
Additionally, it includes a specialized knowledge domain for testing RAG (Retrieval-Augmented Generation) capabilities, allowing agents to search unstructured documents to resolve customer issues.
Who it’s for
- Agent Developers: Those building conversational AI for customer service who need to verify their agent's adherence to policy and tool-use accuracy.
- AI Researchers: People studying the reliability of LLM-based agents in tool-heavy, multi-turn environments.
- RL Practitioners: Developers using the provided Gymnasium-compatible interface to train agents via reinforcement learning.
Highlights
- Multimodal Evaluation: Supports both text-based and end-to-end audio-native voice interactions.
- Diverse Real-World Domains: Pre-built scenarios for airline, retail, telecom, and banking industries.
- Knowledge-Aware Testing: Dedicated RAG pipelines and document search for evaluating knowledge retrieval.
- Rigorous Grading: Includes a refined set of tasks with over 75 fixes to ensure evaluation accuracy and eliminate ambiguous constraints.