modelscope/evalscope

EvalScope – One‑stop evaluation platform for LLMs and multimodal models

What it is – EvalScope is a Python package that lets you run a huge variety of benchmarks on large language models, vision‑language models, embedding / reranker services, and even autonomous agents. With a single CLI command (or a few lines of Python) you can:

• Assess model capability – run classic text benchmarks such as MMLU, C‑Eval, GSM8K, ARC, etc., as well as dozens of newer multimodal, code‑generation, RAG, and agent‑centric suites.
• Stress‑test inference – measure latency, time‑to‑first‑token, throughput, and other performance numbers while the model serves requests.
• Run agents – wrap a benchmark in a multi‑turn AgentLoop that can call tools (bash, python, web search, Docker sandbox, etc.) and record a full trace for each sample.
• Compare models – an interactive web dashboard visualises scores, per‑sample predictions, and performance metrics, supporting pair‑wise “arena” battles.
• Plug‑in anything – you can add your own datasets, models, metrics, or back‑ends (OpenCompass, VLMEvalKit, RAGEval, etc.) with a few configuration lines.

---

Core features (as listed in the README)

Feature	What it gives you
Comprehensive benchmarks	Built‑in support for dozens of industry‑standard suites (MMLU, C‑Eval, GSM8K, MathVista, VQA, code‑benchmarks, RAG, agent benchmarks, etc.).
Multi‑modal & multi‑domain	Works with pure LLMs, vision‑language models, embedding services, rerankers, and generative AIGC models.
Multi‑backend integration	Calls out to OpenCompass, VLMEvalKit, RAGEval, and other third‑party evaluators without you writing glue code.
Agent evaluation mode	Runs benchmarks inside a controllable AgentLoop with pluggable strategies, tools, and Docker sandbox; records a detailed agent_trace.
Inference performance testing	Stress‑testing utilities that report TTFT, TPOT, throughput, and can simulate multi‑turn conversations.
Interactive Web dashboard	React/Vite UI for model comparison, score tables, per‑sample predictions, and visualisation of agent traces.
Arena mode	Pairwise battles between multiple models, automatically ranking them.
Extensibility	Add custom datasets, models, metrics, or even whole new back‑ends via a simple registration API.

---

Typical workflow (quick‑start)

# install the package
pip install evalscope

# evaluate a remote OpenAI‑compatible API (no GPU needed)
# – picks the GSM8K and ARC benchmarks, runs 5 examples each

evalscope eval \
  --model your-model-name \
  --api-url $OPENAI_API_BASE_URL \
  --api-key $OPENAI_API_KEY \
  --eval-type openai_api \
  --datasets gsm8k arc \
  --limit 5

Or, from Python:

from evalscope import run_task, TaskConfig

cfg = TaskConfig(
    model='your-model-name',
    api_url='https://my-endpoint/v1',
    api_key='my_key',
    eval_type='openai_api',
    datasets=['gsm8k', 'arc'],
    limit=5,
)
run_task(cfg)   # same thing as the CLI

The command contacts the model service, runs the selected benchmarks, gathers accuracy scores and performance numbers, and finally launches a local web UI where you can explore the results.

---

Who might use EvalScope?

• Model developers – quickly benchmark a new LLM or VLM against dozens of public datasets and see where it lags.
• Ops / platform teams – run stress‑tests on model serving endpoints to verify latency and throughput SLAs.
• Research labs – evaluate agentic behaviours (tool‑calling, ReAct, code‑generation) with built‑in multi‑turn loops and record traces for analysis.
• Product teams – compare several candidate models side‑by‑side in the dashboard to make a data‑driven selection.

---

Installation & docs

• Install from PyPI: pip install evalscope (requires Python ≥ 3.10).
• Full documentation, installation details, and example notebooks are hosted at https://evalscope.readthedocs.io and a demo website at https://modelscope.github.io/evalscope/.

---

Bottom line – EvalScope is a mature, actively maintained evaluation framework that covers accuracy, speed, and agentic capabilities for a wide spectrum of AI models, all wrapped in a user‑friendly CLI and visual dashboard.