Accio-org/CommerceAgentBench

Commerce Agent Bench (Accio) – Quick Overview

What it is – A open‑source benchmark that measures how well a large‑language‑model (LLM)‑based agent can carry out real‑world, multi‑step commerce workflows. It ships a collection of 107 tasks (CLI, browser, file, API/MCP) that run inside isolated Docker containers and are verified by deterministic or LLM‑assisted checkers.

Why it matters – Most existing LLM evals focus on single‑turn QA or short‑context generation. Commerce Agent Bench flips the script: agents must interact with mock versions of real services (product publishing, freight booking, Shopify admin, etc.), maintain state across calls, and produce the correct end‑state. This makes it a useful proxy for evaluating agents that will be deployed in e‑commerce, supply‑chain, or other business‑process automation settings.

---

Key Features (as described in the README)

Feature	Details
Task diversity	53 CLI, 28 browser, 16 file, 10 API/MCP tasks covering product catalog, logistics, storefront customization, document generation, and more.
Capability slices	65 text‑only, 20 browser‑text‑capable, 22 vision‑required tasks (the latter need image handling).
Stateful evaluation	Each task runs in a fresh container that hosts local mock services replicating SaaS APIs (e.g., Alibaba, FreightOS, Shopify) so agents can perform real‑world state changes without external accounts.
Auditable outputs	After a run the framework saves the full trajectory, verifier results, logs, screenshots, and container metadata for reproducibility.
OpenClaw harness	The benchmark uses the OpenClaw runtime (pinned Docker image acciolyk/accio_bench@sha256:…) which bundles the browser stack, mock services, and the evaluation engine.
Multiple provider routes	Built‑in configs for native Gemini, native Qwen/DashScope, OpenRouter, and generic OpenAI/Anthropic endpoints. Users can also point to custom endpoints.
LLM‑judge	Verification can be LLM‑assisted; the default judge is gemini‑3.1‑pro‑preview.
Reproducibility contract	The repo pins the task set, task definitions, harness version, and runtime image (v1.3.1) to make results comparable across runs.
Leaderboard & showcase	Live leaderboard (commerce-agent-bench.site.accio.ai) and a static UI mock showcase let you explore task screenshots.
Contribution path	Users can contribute new mock services; after review they become part of future benchmark releases.

---

Typical Use Cases

1. Research & model development – Test a new LLM‑agent architecture (e.g., tool‑use planner, memory module) on a realistic commerce workflow suite.
2. Model comparison – Replicate the published tables (Pi, OpenClaw, Accio harness) to see where a model stands relative to Claude Opus, Qwen, Gemini, etc.
3. Pre‑release validation – The Accio team offers to run private evaluations on unreleased checkpoints before you ship a product.
4. Benchmark extension – Contribute a new mock service (e.g., a different marketplace) to keep the benchmark fresh as models improve.
5. Educational demo – Show students how agents can drive a browser, call CLI tools, and manipulate spreadsheets in a controlled environment.

---

Getting Started (from the README)

# 1. Create a Python 3.11+ venv
python3 -m venv .venv && source .venv/bin/activate

# 2. Install the package in editable mode
python -m pip install -e .

# 3. Verify the CLI works
commerce-agent-bench list

Pull the runtime image (required for any run)

docker pull --platform linux/amd64 \
  acciolyk/accio_bench@sha256:1e9cf5c72a56794175b7d06ece036b92e296e6b7e9e9a7fa244026f6acea3859

Run a single task (example with Gemini)

export GEMINI_API_KEY="YOUR_KEY"
commerce-agent-bench run api-amazon-margin-floor-audit \
  --harness openclaw \
  --image acciolyk/accio_bench@sha256:1e9cf5c72a56794175b7d06ece036b92e296e6b7e9e9a7fa244026f6acea3859 \
  --platform linux/amd64 \
  --openclaw-model google/gemini-3.5-flash \
  --openclaw-image-model google/gemini-3.5-flash \
  --openclaw-models-config configs/native_google_direct_models.json \
  --llm-judge-provider gemini \
  --llm-judge-model gemini-3.1-pro-preview \
  --run-id smoke

Run the full suite (or a slice)

commerce-agent-bench run \
  --config configs/openclaw_native_google_direct.yaml \
  --run-id "openclaw-$(date +%Y%m%d-%H%M%S)"

Add --limit 1 for a quick smoke test.

---

Limitations & Open Issues (as disclosed)

Limitation	Explanation
Result bundles not stored in Git	Raw per‑task result files are not version‑controlled yet, so reproducibility relies on the aggregated leaderboard and the pinned runtime image.
Vision tasks require image handling	Only agents that can process screenshots or rendered pages will succeed on the 22 vision‑required tasks; many open‑source agents lack this capability.
Provider‑specific quirks	For Qwen models the “reasoning_content” must be replayed across turns; otherwise scores are not comparable.
Hardware requirement	Docker with linux/amd64 support is mandatory; Apple‑silicon hosts need emulation which can be slower.
No built‑in agents	The repo supplies the benchmark and harness but does not include any ready‑made agent implementations—you must plug in your own model endpoint.
Scoring granularity	Metrics are simple pass‑rate, average steps, time, and token count; they do not capture nuanced quality aspects (e.g., cost efficiency, robustness to UI changes).

---

Who’s Behind It?

Developed and maintained by the Accio team at Alibaba International. The same group also maintains the related Business Arena benchmark, which looks at longer‑term business simulation.

---

Quick Reference Links

• Live leaderboard: https://commerce-agent-bench.site.accio.ai/
• Mock UI showcase: https://commerce-agent-bench-mock-showcase.site.accio.ai/
• Citation file: CITATION.cff
• Contribution guide: CONTRIBUTING.md
• Security policy: SECURITY.md

---

All information above is taken directly from the repository’s README; no external assumptions have been added.