LiveBench/LiveBench

LiveBench: A Challenging, Contamination-Free LLM Benchmark

What it solves

LiveBench addresses the problem of test set contamination in LLM benchmarks, where models may have seen the test questions during their training phase, leading to inflated performance scores. It provides a continuously updating set of challenging questions to ensure that evaluations remain objective and reliable.

How it works

LiveBench limits contamination by releasing new questions monthly and sourcing tasks from recently released datasets, arXiv papers, news articles, and IMDb movie synopses. Unlike many benchmarks that rely on LLM-based judges, LiveBench uses verifiable, objective ground-truth answers that can be scored automatically. It currently covers 18 diverse tasks across six categories: Reasoning, Math, Coding, Language, Data Analysis, and Instruction Following.

Who it’s for

It is designed for AI researchers and developers who need to objectively evaluate the performance of Large Language Models (LLMs) without the risk of contamination.

Highlights

  • Contamination-Free Design: Monthly updates and use of recent real-world data to stay ahead of training sets.
  • Objective Scoring: Uses ground-truth answers for automatic scoring, removing the need for an LLM judge.
  • Diverse Task Suite: Covers a wide range of capabilities including agentic coding tasks (which utilize Docker for execution).
  • Flexible Evaluation Pipeline: Supports OpenAI-compatible APIs, various model providers (Anthropic, Cohere, Mistral, etc.), and parallelized evaluation via tmux.

Related

  • Dispatch
    LiveCodeBench Leaderboard: Contamination-Free Evaluation for Code LLMs

    Hugging Face has introduced the LiveCodeBench leaderboard, a new benchmark developed by researchers from UC Berkeley, MIT, and Cornell to evaluate LLM code generation and reasoning capabilities while preventing benchmark contamination using time-windowed problem sets.

  • Project
    aisa-group/PostTrainBench

    A benchmark that measures the ability of CLI agents to autonomously post-train pre-trained LLMs to improve their performance on specific benchmarks.

  • Project
    MatthewCYM/VoiceBench

    VoiceBench is a benchmarking framework and dataset for evaluating LLM-based voice assistants across tasks like reasoning, safety, and open-ended QA.

  • Project
    groq/openbench

    A provider-agnostic evaluation infrastructure that provides standardized, reproducible benchmarking for LLMs across 95+ evaluation suites.

  • Project
    AudioLLMs/AudioBench

    A universal benchmark for evaluating Audio Large Language Models (AudioLLMs) across 50+ datasets covering speech, audio-scene, and voice understanding.