BigCodeArena: Judging code generations end to end with code executions

Hugging Face has introduced BigCodeArena, the first human-in-the-loop evaluation platform for code generation models that integrates real-time code execution. This platform addresses the limitations of traditional benchmarks by allowing users to run, test, and interact with generated code to determine if it actually works, rather than relying on static code analysis.

The BigCodeArena Evaluation Platform

BigCodeArena extends the Chatbot Arena framework by providing isolated sandbox environments where generated code is automatically executed. This allows users to move beyond reading source code to observing actual program behavior.

Real-Time Execution and Interaction

Users can submit coding tasks and compare two models side-by-side. The platform supports interactive testing, enabling users to click buttons in web apps, play generated games, and edit code to test modifications. It also supports multi-turn conversations, allowing users to refine requirements or request bug fixes over multiple interactions.

Language and Framework Support

The platform currently supports 10 programming languages and 8 execution environments:

  • Languages: Python, JavaScript, TypeScript, HTML, C, C++, Java, Go, Rust, and Markdown.
  • Web Frameworks: React, Vue, and Core Web (vanilla HTML/CSS/JS).
  • Python Frameworks: Streamlit, Gradio, and PyGame.
  • Other: Mermaid for diagrams and general-purpose interpreters/runners for compiled languages.

Community Evaluation Insights

Since February 2025, BigCodeArena has collected over 14,000 conversations from more than 500 users, resulting in 4,700+ high-quality preference votes.

Programming Trends in the Wild

User activity reveals the most common coding scenarios being tested:

  • Web Design (36%): Responsive websites and interactive dashboards.
  • Problem Solving (23%): Algorithms and data structures.
  • Game Development (16%): Interactive games with physics and graphics.
  • Scientific Computing (14%): Data analysis and numerical simulations.
  • Creative Coding (8%) and Diagram Creation (3%).

Python is the most dominant language (4,000+ conversations), followed by JavaScript/TypeScript (3,359) and HTML (1,601).

Model Performance and Elo Rankings

Using the Bradley-Terry model to aggregate pairwise comparisons into Elo ratings, the community identified clear performance tiers:

  • Top Tier: o3-mini and o1-mini consistently lead with the highest Elo ratings and strong robustness across all environments and languages. Claude-3.5-Sonnet follows closely.
  • Mid Tier: GPT-4o, o1, and Gemini-2.0-Pro/Flash form a competitive middle group.
  • Open Source Models: Qwen2.5 variants and Llama-3.3-70B currently lag behind the frontier proprietary models.

Specific strengths were also noted: Gemini-2.0-Pro showed particular strength in Rust, while o3-mini demonstrated the most consistent performance across diverse frameworks like React, Streamlit, and PyGame.

New Benchmarks: BigCodeReward and AutoCodeArena

To supplement crowdsourced data, Hugging Face released two new benchmarks to improve how code quality is judged.

BigCodeReward

BigCodeReward evaluates the ability of LLMs to act as reward models for code. The data shows that providing models with execution results (such as screenshots or logs) significantly increases their alignment with human preferences. For example, GPT-4o's accuracy increased from 54.6% to 63.8% when execution outputs were provided.

AutoCodeArena

AutoCodeArena is an automated benchmark consisting of 600 representative prompts. It uses automated LLM judges (Claude-3.7-Sonnet) to evaluate execution results against a GPT-4.1 baseline. Initial results indicate:

  • GPT-5 establishes a new state-of-the-art by a significant margin.
  • Claude-Opus-4 and Claude-Sonnet-4 form a strong second tier.
  • Qwen3-Coder, Kimi-K2, and GLM-4.5 are the leading open models, narrowing the gap with mid-tier proprietary systems.

Open Source Contributions

In line with the BigCode Project's transparency goals, the following resources have been made public:

  • Codebase: The full evaluation pipelines and Gradio application source are available on GitHub.
  • Crowdsourced Data: 14,000 raw conversations and 4,700 preference votes are available via a Hugging Face Collection.
  • Datasets: The BigCodeReward and AutoCodeArena datasets have been released for research use.

Sources