hkust-nlp/Toolathlon

[ICLR 2026] The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution

What it solves

Toolathlon is a benchmark designed to evaluate how well language agents can use a wide variety of tools to complete complex, long-horizon tasks in realistic software environments. It addresses the need for a standardized way to test an agent's ability to execute multi-step workflows across diverse real-world applications.

How it works

The project provides a comprehensive testing suite featuring over 600 diverse tools. It uses containerization (via Docker or Podman) to create isolated environments for each task, ensuring that agent executions do not interfere with one another. Users can run evaluations through a public service, a local setup, or a decoupled mode where the agent loop runs on the host while the environment remains in a container. It also includes a visualization tool to replay and analyze the reasoning trajectories of the LLMs.

Who it’s for

This tool is primarily for AI researchers and developers building language agents who need to rigorously benchmark their models' tool-use capabilities and long-term planning in realistic scenarios.

Highlights

  • Extensive Toolset: Includes over 600 tools based on real-world software.
  • Long-Horizon Tasks: Focuses on tasks that require multiple sequential tool calls to complete.
  • Isolated Execution: Uses containerized environments to ensure task isolation and parallel evaluation.
  • Flexible Integration: Supports multiple agent scaffolds, including a default implementation and the Claude Agent SDK, with preliminary integration for OpenHands.
  • Trajectory Visualization: Provides a built-in server to replay and visualize LLM reasoning paths.

Related

  • Project
    InternLM/WildClawBench

    An end-to-end agent benchmark that tests AI agents' ability to perform complex, real-world tasks in live environments using tools like browsers and bash terminals.

  • Project
    xlang-ai/OSWorld

    A benchmark and sandboxed environment for evaluating multimodal AI agents on their ability to complete open-ended tasks in real computer operating systems.

  • Project
    benchflow-ai/skillsbench

    A benchmark for evaluating how AI agents leverage and compose modular skills—instructions, scripts, and resources—to perform specialized workflows.

  • Project
    harbor-framework/harbor

    Harbor is a Python framework for running and scaling benchmarks of AI agents and language models. Installable via `pip`/`uv`, it provides a CLI (`harbor run`) that lets you specify a dataset (e.g., Terminal‑Bench‑2.0), an agent (Claude‑Code, OpenHands, etc.), and a model, then executes the benchmark locally or on cloud providers (Daytona, Modal, etc.) with configurable parallelism. It also supports creating custom benchmarks and generating RL roll‑outs.

  • Project
    ByteDance-Seed/EdgeBench

    EdgeBench is a benchmark of 134 real-world tasks designed to evaluate how autonomous AI agents learn and improve through iterative interaction with an environment over long horizons.