facebookresearch/ProgramBench

Can Language Models Rebuild Programs From Scratch?

What it solves

ProgramBench provides a benchmark to evaluate whether AI agents can rebuild a complete codebase from scratch. It tests the ability of models to architect and implement a program that reproduces the original behavior of a compiled binary, using only the binary and its documentation.

How it works

The project provides a set of tests and a framework for AI agents to attempt to recreate programs. Agents are given a compiled binary and documentation, and they must produce a codebase that matches the original program's functionality.

Who it’s for

This tool is designed for researchers and developers working on AI agents, specifically those focusing on software engineering and code generation capabilities.

Highlights

  • A benchmark for rebuilding programs from scratch.
  • Includes a leaderboard to track agent performance.
  • Integrated with the mini-swe-agent baseline.
  • Available as a Python package via pip or uv.

Related