E-Commerce Bench evaluates LLM agents on long‑horizon e‑commerce operations

TL;DR

Qwen introduced E‑Commerce Bench, a year‑long, deterministic simulation of running multiple online stores with ¥100 k capital, and used it to evaluate 18 LLM agents across seven capability axes, showing that even top‑performing models excel in only a subset of dimensions.

Benchmark Overview

  • Goal: Measure an LLM agent’s ability to operate an e‑commerce business continuously, without a natural stopping point.
  • Setup: The agent starts with ¥100 000, may open several stores, and must make daily decisions—researching categories, negotiating with suppliers, pricing, listing, managing inventory, handling promotions, and cash flow—for a simulated 365‑day period.
  • Data source: Real‑world Taobao & Tmall data (6,886 products, 60 categories, 576 suppliers, 12 store types, 10 market events, 8 promotions) desensitized for privacy.
  • Constraints: A daily 600‑minute time budget; each tool call consumes minutes (e.g., balance check = 10 min, store opening = 60 min). Costs are deducted immediately, while revenue is delayed by shipping, escrow, and a 9‑day settlement period.
  • Operational details: Storage fees accrue daily, shipping speed affects freight cost, orders not dispatched within two days are cancelled, and reputation directly multiplies demand.

Deterministic Core Engine

  • Demand model: Fully deterministic; sales volume for a SKU on a given day is computed from base demand, price elasticity, weekend effect, promotions, seasonality, events, store reputation, and a market demand ceiling.
  • Negotiation kernel: Supplier quotes, concessions, and walk‑away decisions are generated by a deterministic kernel, ensuring reproducibility. An LLM only renders the kernel’s decisions into natural language dialogue, preserving the feel of multi‑round bargaining without stochastic price variation.
  • Why deterministic?: Randomness in both buyer and supplier would mask true model differences and enable jailbreak exploits. Freezing economic decisions isolates the agent’s strategic competence.

Evaluation Protocol

  • Runs: Five full‑year episodes per model, totaling 90 episodes across 18 LLM agents (both open‑weight and closed‑source).
  • Primary metric: Year‑end total assets (multiple of the initial ¥100 k). Results ranged from GPT‑5.6 Sol achieving ¥1.43 M (14.31×) to Qwen‑3.5‑Plus averaging ¥1 100.
  • Failure mode: Ten episodes (≈11 %) ended in bankruptcy, typically caused by early over‑stocking and cash‑flow collapse.
  • Secondary axes (scored 0–1 unless otherwise noted):
    1. Negotiation quality – ability to push price below the supplier’s opening quote.
    2. Fraud avoidance – proportion of procurement spend reaching known fraudulent suppliers.
    3. Cash‑flow & solvency – maintenance of liquidity throughout the year.
    4. Operational efficiency – profit per tool‑call.
    5. Operations execution – effective use of actions such as shipping, cash withdrawal, and inventory listing.
    6. Learning over the horizon – measured by AnchorRatio, the degree to which repeat purchase prices improve relative to a random baseline.

Key Findings

Profit Disparities

  • Closed‑source models dominate the profit leaderboard; the best open‑weight model (Qwen‑3.8‑Max‑Preview) achieved a 4.16× return.
  • Asset trajectories fall into three patterns: steady growth (top models), early bankruptcy (mid‑range failures), and flat‑line performance near the initial capital (lower tier).

Negotiation Quality

  • No model reached the supplier’s cost floor. Claude Opus 4.7 scored the highest (0.811), while Kimi K2.6 barely improved over the opening quote (0.596).
  • Performance variation across the six supplier behavior styles was small for a given model but large between models.

Fraud Avoidance

  • Fraudulent spend varied 160‑fold across models, from 0.12 % (Claude Opus 4.7) to 20.11 % (Qwen‑3.5‑Plus).
  • All models performed better than a naïve “no screening” baseline (26.4 % of suppliers are fraudulent). The main differentiator was order conversion from fraudulent contacts, not the number of contacts.

Operational Efficiency

  • Profit per tool call ranged from ¥363 (GPT‑5.6 Sol) to ¥479 (Fable5). Fable5 generated comparable profit with 59.9 % fewer calls.
  • The majority of calls (71.8 %) involved low‑impact actions (shipping, cash withdrawal, inventory listing), while only 6 % affected procurement cost.

Long‑Horizon Learning

  • AnchorRatio quantifies whether repeat purchase prices improve over time. The median across models was 1.369 (worse than random), and only Qwen‑3.8‑Max‑Preview showed genuine learning (AnchorRatio = 0.834).
  • Most models did not reduce purchase prices over the year; supplier choice also drifted toward more fraudulent partners later in the simulation.

Implications for LLM Agent Development

  • Single‑metric evaluation is misleading: A model that maximizes profit may rank near the bottom on fraud avoidance or learning.
  • Deterministic benchmarking provides reproducible, comparable results and isolates strategic competence from stochastic noise.
  • Multi‑axis scoring highlights specific weaknesses (e.g., negotiation, fraud screening) that developers can target for improvement.
  • Room for progress: Even the best scores on each axis fall far short of perfect (1.0), indicating substantial headroom for future LLM agents in autonomous business operations.

Future Directions

  • Extending the deterministic kernel concept to other long‑horizon domains (e.g., supply‑chain logistics, financial trading) could standardize reproducible benchmarks.
  • Adding richer supplier behavior styles and dynamic market shocks would test robustness further.
  • Open‑sourcing the benchmark environment would enable community‑driven model improvements and cross‑lab comparisons.

Citation

@misc{fan2026ecommercebenchevaluatingllm,
  title = {E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation},
  author = {Wei Fan and Xinjie Shen and Xudong Guo and Jianhong Tu and Yang Su and Yinger Zhang and Lianghao Deng and Fengyu Wang and Baohua Dong and Yangqiu Song and Dayiheng Liu},
  year = {2026},
  eprint = {2608.30730},
  archivePrefix = {arXiv},
  primaryClass = {cs.LG},
  url = {https://arxiv.org/abs/2608.30730}
}

All figures referenced in the original blog post are reproduced here conceptually; the benchmark code and data are available via the linked Qwen GitHub repository.

Sources