Accio-org/CommerceAgentBench

CommerceAgentBench: Benchmarking Long-Horizon Agents in High-Fidelity, Stateful, and Reproducible Replicas of Real Online Services

Commerce Agent Bench – What It Is

Commerce Agent Bench is a benchmark suite created by the Accio team at Alibaba International for evaluating autonomous AI agents that need to perform real‑world, multi‑step commerce workflows. Unlike typical question‑answer or single‑turn tests, this benchmark runs each task inside an isolated Docker container that hosts mock versions of real‑world services (e.g., Alibaba product publishing, Freightos logistics, Shopify admin). The agent must interact with browsers, command‑line tools, APIs, spreadsheets, and documents just as a human would, and the outcome is verified by a deterministic or LLM‑assisted grader.

Key Features (as described in the README)

Feature Details
Task set 107 tasks covering CLI, browser, file, and API/MCP interactions. They are split into three capability slices: 65 text‑only, 20 browser‑text‑capable, and 22 vision‑required tasks.
Stateful evaluation Each task runs in a fresh container with local mock services that emulate SaaS, commerce, messaging, and document systems, so the agent actually changes state (e.g., creates a product listing).
Auditable outputs After a run the framework saves the full configuration, execution trajectory, verifier results, logs, and container metadata, enabling reproducibility checks.
Harness The benchmark uses the OpenClaw runner (version 2026.5.22) packaged in a pinned Docker image (acciolyk/accio_bench@sha256:…).
Verification Every task has its own grader; some graders are pure deterministic checks, others use an LLM judge (default: Gemini 3.1‑pro‑preview). A task passes only when all required checks succeed.
Metrics reported Pass rate (tasks passed / 107), average steps (tool‑call count), average time (wall‑clock), and average tokens (model usage).
Reference results The README includes a snapshot leaderboard for three harnesses (Pi, OpenClaw, Accio) showing how a dozen model families performed (e.g., Claude Opus 5 tops with ~61 % pass).
Extensibility Users can contribute new mock services; contributed services are first placed under mock_services/contrib/ and later merged into the official task set.
Quick start Install with a Python 3.11+ virtualenv, pull the pinned Docker image, then run a single task or the full suite via the commerce-agent-bench CLI.
Reproducibility contract The repository pins the task set, task definitions, harness version, and runtime image (v1.3.1). Users are asked to report provider, model, judge, and other runtime details to make results comparable.

Who Might Use It?

  • Research labs building LLM‑driven agents that need to act on web UIs, CLIs, or APIs.
  • LLM providers who want an objective, stateful benchmark to compare model families on commerce‑oriented workflows.
  • Product teams evaluating internal or pre‑release agents before deploying them to real e‑commerce platforms.

How to Get Started (summary of the README steps)

  1. Set up the environment – Docker (Linux/AMD64) and Python 3.11+. Create a virtualenv and install the package (pip install -e .).
  2. Pull the benchmark runtime – docker pull acciolyk/accio_bench@sha256:<digest> (the digest is pinned in the repo).
  3. Provide API keys – Export keys for the model you want to test (e.g., GEMINI_API_KEY) and for the LLM judge (GEMINI_API_KEY for Gemini‑3.1‑pro‑preview, or the appropriate key for OpenAI/Anthropic, etc.).
  4. Run a task – Example command runs the Amazon margin floor audit task with Gemini 3.5‑flash.
  5. Run the full suite – Use commerce-agent-bench run --config configs/openclaw_native_google_direct.yaml (optionally limit with --limit 1 for a smoke test).
  6. Inspect results – After the run, the runs/<run_id>/ directory contains summary.json, report.html, and per‑task manifests with logs, screenshots, and output artifacts.

License & Community

The project is open‑source (the README does not list a specific license, but the repository includes standard contribution and security policies). Contributions are welcomed, especially new mock services that expand the benchmark’s coverage. The team offers to evaluate private models on request and is open to collaborations.

Bottom line: Commerce Agent Bench is a genuine, production‑grade benchmark for testing the ability of LLM‑powered agents to carry out end‑to‑end commerce tasks in a reproducible, auditable way.

Related

  • Project
    InternLM/WildClawBench

    An end-to-end agent benchmark that tests AI agents' ability to perform complex, real-world tasks in live environments using tools like browsers and bash terminals.

  • Project
    TIGER-AI-Lab/ClawBench

    An open-source benchmark for evaluating AI browser agents on everyday online tasks across live websites, measuring end-to-end success using a recording pipeline and agentic evaluator.

  • Project
    AISBench/benchmark

    A model evaluation tool based on OpenCompass that provides accuracy and performance benchmarking for both local and service-based AI models across text and multimodal tasks.

  • Project
    harbor-framework/terminal-bench

    A continuous benchmark for measuring the frontier of AI agent capabilities using a diverse and difficult set of evolving terminal-based tasks.

  • Project
    aisa-group/PostTrainBench

    A benchmark that measures the ability of CLI agents to autonomously post-train pre-trained LLMs to improve their performance on specific benchmarks.