EvolvingLMMs-Lab/lmms-eval

One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks

lmms‑eval – A Unified Evaluation Toolkit for Multimodal LLMs

What it is – lmms-eval is a Python library that lets researchers and engineers run reproducible, efficient, and statistically‑sound benchmarks on multimodal large language models (vision‑language, audio‑language, video‑language, etc.). It ships with a growing catalog of >100 tasks (MMMU, MME, VideoMME, MathVista, …) and wrappers for 30+ model families such as Qwen‑2.5‑VL, Qwen‑3‑VL, LLaVA‑OneVision, InternVL‑2, VILA, and any OpenAI‑compatible API.

Why it matters – Multimodal evaluation is currently fragmented: datasets live in different places, preprocessing varies, and results are often reported without confidence intervals. lmms‑eval solves this by providing a single, deterministic pipeline that:

  • Guarantees reproducibility (same model + same task → identical numbers).
  • Maximises throughput with async serving, adaptive batching, and video I/O optimisations (up to ~3.5× faster than earlier versions).
  • Emits trustworthy statistics (confidence intervals, paired‑test p‑values, clustered standard errors) so you can tell whether a gain is real.

Key features

Feature What it does
Unified task catalog 100+ YAML‑defined benchmarks covering image, video, and audio modalities.
Model back‑ends Native wrappers for chat‑style models, legacy simple models, vLLM, SGLang, and any OpenAI‑compatible endpoint.
Chat‑style API Structured ChatMessages (role + multimodal content) – supports interleaved image/video/audio in a single request.
Statistical reporting Confidence intervals, paired‑t tests, standard‑error clustering, per‑sample token counts, throughput metrics.
Evaluation‑as‑a‑service Stand‑alone HTTP server that queues jobs, runs them on dedicated GPUs, and returns results via a REST API.
Web UI Optional React‑based UI for selecting models/tasks, previewing commands, streaming live output, and browsing logs.
Async/Sync clients Python client libraries (EvalClient, AsyncEvalClient) to submit jobs from training loops without blocking.
Extensible Add new models by implementing generate_until and new tasks via a YAML config plus a doc_to_messages function.
Multi‑language docs README and full documentation available in 17 languages.

Quick start (run a tiny test in < 5 min)

git clone https://github.com/EvolvingLMMs-Lab/lmms-eval.git
cd lmms-eval && uv pip install -e "[all]"
python -m lmms_eval \
  --model qwen2_5_vl \
  --model_args pretrained=Qwen/Qwen2.5-VL-3B-Instruct \
  --tasks mme \
  --batch_size 1 \
  --limit 8

If you see a JSON‑L log with accuracy numbers, the toolkit is ready.

Installation notes

  • The project recommends the uv package manager for deterministic environments (uv install -e "[all]").
  • A classic pip install git+https://github.com/EvolvingLMMs-Lab/lmms-eval.git also works.
  • For caption‑style datasets you’ll need Java 8 for the pycocoeval API.

Typical workflow

  1. Pick a model – use a built‑in wrapper (qwen2_5_vl, internvl_2, etc.) or point to an OpenAI‑compatible endpoint.
  2. Select tasks – any of the 100+ tasks listed in docs/advanced/current_tasks.md.
  3. Run evaluation – via the CLI (python -m lmms_eval …), the HTTP server, or the Web UI.
  4. Analyse – results are emitted as flattened JSONL files; you can compute aggregate metrics, confidence intervals, or feed them into downstream dashboards.

Extending the library

  • New model – subclass lmms_eval.api.model.lmms, set is_simple=False, implement generate_until that builds a ChatMessages object and calls the model’s chat template.
  • New benchmark – drop a YAML file under lmms_eval/tasks/ describing the dataset path, split, and a doc_to_messages function that converts each record into the structured chat format.
  • Custom metrics – plug a process_results function and list the metric names in the YAML.

Resources

All statements above are taken directly from the repository’s README; no external assumptions have been added.

Related

  • Project
    huggingface/lighteval

    A comprehensive toolkit from Hugging Face for evaluating LLMs across multiple backends with support for over 1,000 benchmarks and custom metrics.

  • Project
    open-compass/VLMEvalKit

    An open-source evaluation toolkit for large vision-language models that enables one-command evaluation across 70+ benchmarks and 200+ models.

  • Project
    AISBench/benchmark

    A model evaluation tool based on OpenCompass that provides accuracy and performance benchmarking for both local and service-based AI models across text and multimodal tasks.

  • Project
    alexzhang13/rlm

    An inference engine and training environment for Recursive Language Models (RLMs) that enable LLMs to handle near-infinite context by programmatically decomposing tasks via recursive sub-calls in a REPL environment.

  • Project
    relari-ai/continuous-eval

    continuous-eval is a Python library for modular, data‑driven evaluation of LLM‑powered pipelines. It provides a catalog of retrieval, generation, code, and agent metrics, lets you attach those metrics to individual pipeline modules, runs them on a dataset, and can enforce regression tests. Custom LLM‑as‑a‑judge metrics are also supported.