ByteDance-Seed/EdgeBench

What it solves

EdgeBench addresses the limitation of one-shot performance benchmarks for AI agents. Instead of measuring if an agent can solve a task immediately, it evaluates an agent's ability to learn and improve through iterative interaction with a real-world environment over an extended period (up to 12+ hours per task).

How it works

EdgeBench consists of 134 real-world tasks across six categories (Scientific & ML, Systems & SE, Optimization, Knowledge, Formal, and Games). It uses a specialized evaluation harness called SForge, which employs a two-container isolation system: a "work" container where the agent operates and a hidden "judge" container that runs tests. Agents receive granular feedback on their submissions and can iterate on their solutions in a closed loop until the time budget expires. The final score is based on the best result achieved during the entire trajectory.

Who it’s for

It is designed for AI researchers and developers building autonomous agents who want to measure how their models scale and improve when given time to interact with executable environments and receive feedback.

Highlights

• Long-horizon evaluation: Tracks improvement over 12+ hours per task rather than a single attempt.
• Two-container isolation: Prevents agents from "hacking" the evaluation by separating the work and judge environments.
• Diverse task set: Includes 134 tasks, with 51 publicly released, covering complex domains like formal verification and systems engineering.
• Flexible backend support: SForge supports local Docker, Kubernetes for large-scale runs, and E2B cloud sandboxes.