LiveCodeBench Leaderboard: Contamination-Free Evaluation for Code LLMs

Hugging Face has launched the LiveCodeBench leaderboard, a new evaluation framework developed by researchers from UC Berkeley, MIT, and Cornell. This benchmark is designed to provide a holistic and contamination-free measurement of Large Language Model (LLM) coding capabilities by using problems collected over time from competitive programming platforms.

Preventing Benchmark Contamination via Temporal Evaluation

LiveCodeBench addresses the critical issue of benchmark contamination—where models are trained on the test data they are later evaluated against—by annotating every coding problem with its release date. This allows researchers to implement a "scrolling over time" strategy: for a model with a training cutoff date D, the benchmark can compute scores using only problems released after date D. This ensures that the model is evaluated on truly unseen problems, providing a more accurate measure of generalization rather than memorization.

Holistic Evaluation Scenarios

Unlike traditional benchmarks that focus solely on code generation, LiveCodeBench utilizes problems curated from LeetCode, AtCoder, and CodeForces to assess four distinct coding scenarios. All scenarios are evaluated using the Pass@1 metric, which calculates the ratio of correct answers to total attempts.

Code Generation

Models are provided with a natural language problem statement and example tests. The goal is to generate a correct solution, which is then evaluated based on functional correctness using a set of hidden test cases.

Self Repair

This scenario tests a model's ability to iterate on its own code. If a model generates an incorrect program, it is provided with error feedback—such as an exception message or a failing test case—and must generate a fix to achieve functional correctness.

Code Execution

Models are tasked with predicting the output of a specific program snippet (a function) when given a test input. Correctness is determined by whether the model's predicted output matches the actual result of executing the function with that input.

Test Output Prediction

In this scenario, models are given a problem statement and a test case input. They must generate the expected output for that input without seeing the function's implementation. This is evaluated using an exact match checker.

Key Findings and Model Performance

Initial evaluations reveal that model rankings vary depending on the specific coding scenario, suggesting that different models possess different strengths in reasoning and execution.

• GPT-4-Turbo: This model is the best performer across most scenarios. It shows a particularly strong margin in self-repair tasks, indicating a high capability for utilizing compiler feedback.
• Claude-3-Opus: This model outperforms GPT-4-Turbo specifically in the test output prediction scenario, suggesting superior natural language reasoning capabilities.
• Mistral-Large: This model demonstrates significantly better performance on natural language reasoning tasks, including code execution and test output prediction.

Submission and Contribution

The LiveCodeBench framework is open-source, with the dataset and code available on Hugging Face and GitHub. Developers can evaluate their models using the lcb_runner module and submit their results via an official form to be included in the leaderboard.