EleutherAI/lm-evaluation-harness

A framework for few-shot evaluation of language models.

What it solves

This project provides a unified framework for testing generative language models across a vast array of evaluation tasks. It eliminates the need to run separate, fragmented benchmarks for different models, allowing researchers and developers to compare LLM performance consistently and reproducibly using standard academic benchmarks.

How it works

The harness acts as a standardized interface between various model backends and a library of evaluation tasks. It supports a wide range of model loading methods, including Hugging Face transformers, vLLM, NVIDIA NeMo, Megatron-LM, and commercial APIs (like OpenAI). The framework allows users to specify a model, a set of tasks (e.g., hellaswag), and configuration options via a CLI or Python API, then automatically handles the prompt generation, model inference, and metric calculation.

Who it’s for

It is designed for AI researchers, LLM developers, and organizations (such as NVIDIA, Cohere, and Mosaic ML) who need to rigorously evaluate the capabilities of their models against industry-standard benchmarks to ensure comparability and with the Open LLM Leaderboard.

Highlights

  • Extensive Benchmark Library: Includes over 60 standard academic benchmarks with hundreds of subtasks.
  • Broad Model Support: Compatible with Hugging Face, vLLM, Megatron-DeepSpeed, NVIDIA NeMo, and commercial APIs.
  • Flexible Parallelism: Supports data-parallel evaluation, model sharding (via accelerate), and native PyTorch Tensor Parallelism.
  • Customizability: Allows for custom prompts, evaluation metrics, and YAML-based task configuration.
  • Multimodal Prototyping: Experimental support for text+image multimodal input tasks.
  • Industry Standard: Serves as the backend for the Hugging Face Open LLM Leaderboard.

Related

  • Project
    huggingface/lighteval

    A comprehensive toolkit from Hugging Face for evaluating LLMs across multiple backends with support for over 1,000 benchmarks and custom metrics.

  • Project
    groq/openbench

    A provider-agnostic evaluation infrastructure that provides standardized, reproducible benchmarking for LLMs across 95+ evaluation suites.

  • Project
    PacificAI/langtest

    A comprehensive evaluation library for NLP and LLM models to test for robustness, bias, fairness, and safety.

  • Dispatch
    Hugging Face Open LLM Leaderboard MMLU Evaluation Analysis

    Hugging Face explains how different implementation details in the MMLU benchmark lead to significantly different performance scores and rankings for the same LLMs.

  • Project
    open-compass/opencompass

    A comprehensive evaluation platform for large language and vision-language models that provides a unified framework for benchmarking across diverse datasets and paradigms.