EvolvingLMMs-Lab/lmms-eval

One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks

What it solves

LMMs-Eval provides a unified, reproducible, and efficient toolkit for evaluating Large Multimodal Models (LMMs). It addresses the fragmentation in the multimodal evaluation ecosystem, where inconsistent post-processing and scattered datasets often lead to unreliable or non-reproducible results across different teams.

How it works

The toolkit implements a standardized pipeline for probing model intelligence across image, video, and audio modalities. It supports a wide range of model backends (including vLLM, SGLang, and OpenAI-compatible APIs) and provides a structured protocol for transforming dataset documents into chat messages. It also offers an HTTP evaluation server to decouple evaluation from training, allowing for asynchronous job submission and remote access.

Who it’s for

It is designed for AI researchers and model developers who need to benchmark frontier multimodal models against a diverse set of tasks to ensure improvements are real and statistically grounded.

Highlights

  • Extensive Support: Includes over 100 evaluation tasks and support for 30+ model families.
  • Multimodal Capabilities: Comprehensive evaluation for image, video, and audio modalities.
  • High Efficiency: Features async serving, adaptive batching, and TorchCodec-optimized video I/O for maximum GPU saturation.
  • Statistically Grounded: Provides confidence intervals, clustered standard errors, and paired t-tests to ensure results are trustworthy.
  • Interactive Tools: Includes a Web UI for configuration and a production-ready HTTP server for remote evaluation workflows.

Related

  • Project
    huggingface/lighteval

    A comprehensive toolkit from Hugging Face for evaluating LLMs across multiple backends with support for over 1,000 benchmarks and custom metrics.

  • Project
    confident-ai/deepeval

    DeepEval is an open‑source Python framework that lets you write pytest‑style tests for LLM‑powered apps (agents, chatbots, RAG pipelines). It ships >30 built‑in metrics—including LLM‑as‑a‑judge scores like G‑Eval and JevEval—as well as agent‑, RAG‑, multimodal‑, and bias‑related checks. Metrics run locally or with any LLM you choose. The library integrates with LangChain, CrewAI, OpenAI, Anthropic, AWS AgentCore, Google ADK, LlamaIndex, and even the TypeScript AI SDK, providing decorators and callbacks to capture full execution traces. A simple CLI (`deepeval test run …`) works in CI/CD, and the optional Confident AI platform adds cloud reporting and governance. Install with `pip install -U deepeval`, write `LLMTestCase` objects, pick a metric, and run—making LLM quality testing as systematic as traditional software testing.

  • Project
    Yinsongxu/LLM2Jev

    A framework that converts local LLMs and VLMs into structured decision models by extracting probabilities from logits during prefill, bypassing token-by-token decoding for lower latency.

  • Project
    open-compass/VLMEvalKit

    An open-source evaluation toolkit for large vision-language models that enables one-command evaluation across 70+ benchmarks and 200+ models.

  • Dispatch
    BigCodeBench: A New Benchmark for Complex Python Code Generation

    Hugging Face has released BigCodeBench, a benchmark of 1,140 function-level tasks designed to evaluate LLMs on practical programming and diverse library usage, addressing the simplicity and contamination issues of HumanEval.