CUGA Harness Outperforms Hermes and Deep Agents on AppWorld Benchmark

TL;DR

IBM’s open‑source CUGA harness completed 81% of AppWorld benchmark tasks with the same GLM‑5.3‑flash model, far outpacing Hermes (47%) and Deep Agents (30%). The result shows that for enterprise automation the harness – not just the model – determines reliability, rule enforcement, and deployability.

The experiment proves the harness matters

A controlled test ran three popular agent harnesses – CUGA, Hermes, and Deep Agents – on the same open‑weight model (GLM‑5.3‑flash) across 90 AppWorld tasks (30 multi‑app scenarios). All runs used the Harbor evaluation framework and identical task inputs; only the harness configuration differed. The outcome:

Harness Task‑completion (TGC) Scenario‑completion (SGC)
CUGA 81.1 % (73/90) 66.7 % (20/30)
Hermes 46.7 % (42/90) 26.7 % (8/30)
Deep Agents 30.0 % (27/90) 6.7 % (2/30)

The SGC metric, which requires all three variants of a scenario to succeed, emphasizes consistency; CUGA achieved more than three times the scenario success of the next best harness.

"The most useful number is SGC. A scenario passes only when all three of its variants pass, so SGC rewards consistency, not a lucky run. Here the gap is larger: 20 scenarios for CUGA, 8 for Hermes, and 2 for Deep Agents." – IBM Research blog

Why CUGA excels: key harness features

CUGA’s architecture adds a compact, repeatable loop that offloads decision‑making from the model to the harness:

  • Task decomposition – splits a multi‑app request into single‑app steps, reducing combinatorial tool selection.
  • Focused tool sets – each step sees only the tools relevant to its app plus a few shared utilities, avoiding the noise of 469 total tools.
  • Tool short‑lister – dynamically discovers additional tools a step may need.
  • CodeAct & sandbox – the model emits Python that calls APIs; the code runs in a time‑limited, restricted environment.
  • Pre‑execution verification – every code block that mutates state is reviewed before execution.
  • Automatic reflection – after each step the harness audits results, corrects the plan, and flags pagination issues.

These enforcement points dramatically cut erroneous decisions and surface mistakes early, leading to higher overall success.

Enforcing business rules beyond prompts

Enterprise agents must obey strict policies (e.g., payment limits, PII handling). CUGA moves such rules from fragile system prompts into a policy layer that the harness enforces at four execution points:

Run point Policy type Enforcement action Example
User intent Intent Guard Block or answer before any agent action Reject a request to delete all customer records
Agent guidance Playbook Provide a standard procedure Follow a step‑by‑step refund workflow
Tool execution Tool Approval Pause for human sign‑off, show generated code Require manager approval before a payment tool runs
Tool execution ToolGuard Hard check that aborts disallowed calls Block any payment > $1,000
Final response Output Formatter Shape answer to an approved template Use a compliance‑approved reply format

The Policy Playground demo shows the same user message producing two different outcomes: an unrestricted agent pays $90 (including a denied claim), while the policy‑enabled agent holds the payment for approval and only pays the authorized $10.

"For "Pay everything I owe.", the agent without policies paid $90, including a denied claim in collections. The agent with policies held the payment for approval, then paid only the $10 that was due." – IBM Research blog

When CUGA is the right choice

CUGA is designed for enterprise automation where:

  • Rules must be enforced at runtime, not merely suggested in prompts.
  • Deployment must stay inside a sovereign boundary (private cloud, air‑gapped clusters, Kubernetes, OIDC auth).
  • Transparency of costs (token usage, tool calls, runtime) is required.
  • Teams prefer configuration over code – policies, budgets, and tool sets are set via UI or SDK and versioned.

CUGA has also topped public leaderboards with hosted models (WebArena 2025‑2025, AppWorld 2025‑2026), demonstrating that its advantages persist beyond open‑weight models.

When to consider other harnesses

CUGA is not a universal solution. Choose alternatives for specialized workloads:

  • Terminal‑coding agents – use opencode, Codex, or Claude Code.
  • Always‑on personal assistants – Hermes (Nous Research) excels in chat‑centric contexts.
  • Blank‑canvas graph workflows – LangGraph provides low‑level flexibility; CUGA itself is built on LangGraph.

How to try CUGA yourself

The IBM Research team invites the community to submit alternative harness configurations for the same AppWorld tasks; results will be re‑run and published transparently.

Limitations of the reported study

  • The comparison covers only 90 tasks from a single benchmark, so findings may not extrapolate to all enterprise scenarios.
  • Harness setups differed in detail (e.g., tool short‑listing, middleware), which could influence performance.
  • No policy enforcement was active during the benchmark, so the measured gains stem solely from orchestration logic, not rule checking.

Bottom line

Selecting the right agent harness is as critical as picking the model itself. In a head‑to‑head test with identical models, CUGA’s enterprise‑focused design delivered more than double the task‑completion rate of Hermes and almost three times that of Deep Agents, proving that harness‑level orchestration, verification, and policy enforcement are decisive factors for reliable business‑grade AI agents.

Sources