BigCodeBench: A New Benchmark for Complex Python Code Generation

Hugging Face has introduced BigCodeBench, a new benchmark designed to evaluate the practical programming capabilities of large language models (LLMs) by focusing on complex, real-world tasks. Unlike previous benchmarks like HumanEval, which often rely on simple algorithm-oriented snippets, BigCodeBench requires models to compose multiple function calls across 139 different libraries to solve user-oriented problems.

Addressing Limitations of Existing Benchmarks

BigCodeBench was developed to solve three primary issues found in existing code evaluation frameworks:

  • Task Simplicity: HumanEval tasks are often too simple to represent real-world software development, which typically requires the integration of diverse libraries.
  • Contamination and Overfitting: There are growing concerns that LLMs have been trained on HumanEval data, making it an unreliable measure of generalization.
  • Lack of General-Purpose Tools: Existing alternatives are often too domain-specific, deterministic, or focused exclusively on agents (e.g., DS-1000 or SWE-bench), leaving a gap for an easy-to-use, broad programming benchmark.

Technical Design and Task Structure

BigCodeBench consists of 1,140 function-level tasks. To ensure rigorous evaluation, each task includes an average of 5.6 test cases with an average branch coverage of 99%.

Task Variants

Two versions of the benchmark exist to test different model capabilities:

  • BigCodeBench-Complete: A code completion scenario where LLMs finish a function based on detailed instructions in a docstring.
  • BigCodeBench-Instruct: A more challenging variant where requirements are presented in a conversational, less verbose manner to evaluate instruction-tuned LLMs.

Task Creation Process

Tasks are generated through a "Human-LLM collaboration process":

  1. Seeding: The process began with the ODEX dataset (Stack Overflow Python one-liners).
  2. Expansion: GPT-4 expanded these one-liners into comprehensive function-level tasks.
  3. Refinement: 20 human experts with over 5 years of Python experience guided GPT-4 in a sandbox to refine tasks and add test cases.
  4. Verification: 7 additional human experts cross-checked the quality. Human experts achieved an average performance of 97% on the sampled tasks.

LLM Performance and Evaluation Metrics

Evaluation is conducted using Pass@1 with greedy decoding, measuring the percentage of tasks correctly solved on the first attempt. To combat "model laziness"—where instruction-tuned models omit essential import statements in long prompts—the researchers use calibrated Pass@1, which adds missing setups (imports and global constants) during evaluation.

Key Findings

  • Performance Gap: LLMs perform significantly lower than humans. GPT-4o is currently the top-performing model, achieving a calibrated Pass@1 of 61.1% on BigCodeBench-Complete and 51.1% on BigCodeBench-Instruct.
  • Closed vs. Open Models: There is a notable performance gap between closed-source and open-source LLMs.
  • Difficulty: The benchmark is highly challenging; on BigCodeBench-Complete, 149 tasks remain unsolved by all tested models, while only 6 tasks were solved by every model.

Ranking via Elo Rating

To provide a more granular comparison than Pass@1, the team adapted the Elo rating system (commonly used in chess). In this system, each task is treated as a game and each model as a player. GPT-4o leads the rankings by a large margin, followed by DeepSeekCoder-V2 in the second tier.

Evaluation Framework and Implementation

BigCodeBench provides a user-friendly evaluation framework available via PyPI (pip install bigcodebench). The framework is based on EvalPlus but adapted for unittest and the diverse library dependencies of BigCodeBench.

Workflow

  1. Generation: Use bigcodebench.generate to create code samples.
  2. Sanitization: Use bigcodebench.sanitize to remove natural language and ensure the code is compilable.
  3. Evaluation: Use bigcodebench.evaluate within a Docker sandbox to execute the code against test cases.

Future Roadmap

The BigCode project aims to evolve the benchmark in several key areas:

  • Multilingualism: Expanding beyond Python to other programming languages.
  • Rigorousness: Improving test case augmentation to ensure all LLM-generated solutions are correctly assessed.
  • Generalization: Testing models on emerging libraries like transformers and langchain.
  • Evolution: Periodically updating the benchmark to account for deprecated library versions and evolving training data.
  • Interaction: Moving toward "LLMs as Agents" by allowing models to interact with browsers and terminals for self-debugging and reflection.

Sources