ByteDance-Seed/EdgeBench

What it solves

EdgeBench addresses the limitation of one-shot performance benchmarks for AI agents. Instead of measuring a single attempt, it evaluates an agent's ability to learn and improve through iterative interaction with real-world environments over long periods (up to 12+ hours per task), tracking the trajectory of improvement rather than just the final result.

How it works

EdgeBench consists of 134 real-world tasks across six categories (Scientific & ML, Systems & SE, Optimization, Knowledge, Formal, and Games), with 51 tasks publicly released. It uses a specialized evaluation harness called SForge, which employs a two-container isolation system: a "work" container where the agent operates and a hidden "judge" container where tests are executed. Agents receive granular feedback on their submissions and can iterate in a closed loop until a timeout is reached.

Who it’s for

It is designed for researchers and developers building autonomous AI agents who want to measure how their models learn from environment feedback and scale their performance over time.

Highlights

• Long-horizon evaluation: Tracks agent progress over 12+ hours per task.
• Two-container isolation: Prevents evaluation hacking by separating the work and judge environments.
• Scaling law discovery: Found that agent performance follows a log-sigmoid scaling law relative to interaction time.
• SForge Harness: Provides a scalable framework with Kubernetes backend support for running large-scale agent evaluations.