open-compass/opencompass

OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.

What it solves

OpenCompass provides a unified, fair, and reproducible platform for evaluating large language models (LLMs) and large vision-language models (LVLMs). It eliminates the need to manually run multiple disparate benchmarks by offering a one-stop system to assess model quality and effectiveness across various dimensions.

How it works

OpenCompass uses a modular design that allows users to integrate various models (via HuggingFace, API, or custom interfaces) and datasets. It supports multiple evaluation paradigms, including zero-shot, few-shot, and chain-of-thought, and can be configured via CLI or Python scripts. For efficiency, it implements distributed evaluation to handle billion-scale models and integrates with acceleration backends like vLLM and LMDeploy.

Who it’s for

It is designed for AI researchers and practitioners who need to benchmark their models against industry standards, compare open-source and proprietary API models, or develop new evaluation benchmarks.

Highlights

  • Broad Compatibility: Supports 20+ models and 70+ datasets with approximately 400,000 questions.
  • Distributed Evaluation: Enables large-scale evaluation tasks to be completed in a few hours through task division.
  • Flexible Paradigms: Supports zero-shot, few-shot, and CoT evaluations with customizable prompt templates.
  • LLM-as-Judge: Includes tools like GenericLLMEvaluator for using LLMs to judge other model outputs.
  • Multimodal Support: Integrates with VLMEvalKit for native multimodal dataset loading and evaluation.

Related

  • Project
    open-compass/VLMEvalKit

    An open-source evaluation toolkit for large vision-language models that enables one-command evaluation across 70+ benchmarks and 200+ models.

  • Project
    modelscope/evalscope

    EvalScope is a Python‑based, all‑in‑one evaluation framework for LLMs, vision‑language, embedding, and agent models. It runs dozens of benchmarks, provides stress‑testing (latency, throughput), supports multi‑turn agent loops with tool calling, and visualises results in an interactive web dashboard. Install with `pip install evalscope` and evaluate a model via a single CLI command or a few lines of Python.

  • Project
    groq/openbench

    A provider-agnostic evaluation infrastructure that provides standardized, reproducible benchmarking for LLMs across 95+ evaluation suites.

  • Project
    huggingface/lighteval

    A comprehensive toolkit from Hugging Face for evaluating LLMs across multiple backends with support for over 1,000 benchmarks and custom metrics.

  • Project
    evidentlyai/evidently

    An open-source Python framework to evaluate, test, and monitor ML and LLM-powered systems, helping developers detect data drift and ensure output quality.