ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration

ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration

IBM Research has introduced ScarfBench (Self-Contained Application Refactoring Benchmark), an open benchmark designed to evaluate the ability of AI agents to perform cross-framework migrations in Enterprise Java. This benchmark addresses the critical gap in existing software engineering benchmarks by focusing on the complex requirements of real-world modernization, such as preserving application behavior and managing runtime dependencies, rather than simple code translation.

ScarfBench Framework and Methodology

ScarfBench evaluates AI agents on their ability to migrate applications across three major Java ecosystems: Spring, Jakarta EE, and Quarkus. Unlike traditional benchmarks that compare generated code against a reference implementation, ScarfBench uses a functional validation approach. A migration is only considered successful if the application meets three criteria:

  1. Build Success: The application must build successfully.
  2. Deployment Success: The application must deploy correctly.
  3. Behavioral Validation: The application must pass behavioral tests.

Benchmark Scale and Composition

The benchmark is constructed from a JSR-based enterprise Java taxonomy and includes the following metrics:

Metric Value
Applications 34
Framework implementations 102
Migration tasks 204
Lines of code ~151K
Source and test files ~2,000
Expert-written tests 1,331

Performance of Frontier AI Agents

Evaluation of state-of-the-art coding agents reveals that framework migration remains a significant challenge. The data shows a clear degradation in success rates as the validation criteria become more stringent: compile success is the highest, followed by deploy success, with behavioral success being the lowest.

Key findings from the agent evaluations include:

  • Low Behavioral Success: Even the strongest current agents achieve less than 10% behavioral success, indicating a gap between generating compilable code and preserving actual application behavior.
  • Framework Variability: Migration difficulty varies by target framework, with Jakarta EE being particularly challenging.
  • Agent Overconfidence: Agents often misreport their own success. For example, Claude Code reported successful builds for 29 out of 30 whole applications, but only 22 actually built successfully, while one application it marked as failed actually built correctly.

Technical Challenges in Java Modernization

ScarfBench analysis demonstrates that the primary difficulty in framework migration is not the translation of Java source code, but the management of dependencies across configuration, infrastructure, and runtime environments.

Iterative Dependency Resolution

Migration is an iterative process rather than a linear source-to-source transformation. Agents frequently revisit layers in the following order of frequency:

  • Configuration
  • Web
  • Database
  • Service

Common transitions occur between Configuration $\leftrightarrow$ Web and Service $\leftrightarrow$ Database, suggesting that agents must resolve cascading changes across these layers.

Non-Code Transformation Failures

A significant portion of migration failures are not caused by code errors but by environmental and tooling issues, including:

  • Docker cache inconsistencies
  • Port connectivity problems
  • Maven wrapper and build tooling issues

Conclusion and Resources

While frontier agents can automate parts of the migration process, reliable validation and architectural reasoning remain essential for successful enterprise modernization. ScarfBench provides a standardized way to measure progress toward autonomous application modernization.

Available Resources

Sources