NVIDIA-NeMo/Evaluator

Open-source library for scalable, reproducible evaluation of AI models and benchmarks.

What it solves

NeMo Evaluator provides a standardized framework for testing and measuring the performance of Large Language Models (LLMs) across a wide variety of tasks. It simplifies the process of running benchmarks, managing model interactions, and analyzing results, reducing the need to create custom evaluation scripts for every new model or dataset.

How it works

The framework uses a modular architecture consisting of several key components:

  • Benchmarks & Solvers: It includes 17 built-in benchmarks (covering math, code, safety, and agentic tasks) and supports external harness integrations. "Solvers" determine how the model interacts with the task (e.g., simple chat, tool calling, or agentic execution).
  • Adapter Proxy: A local interceptor proxy sits between the agent and the model. It can cache requests, log token usage, modify payloads, or limit the number of turns in a conversation without requiring changes to the model itself.
  • Sandboxes: For tasks requiring code execution or agentic behavior, the system spins up Docker or SLURM-based sandboxes to safely run and verify the model's output.
  • Reporting & Export: Results are processed into multi-format reports or exported to experiment trackers like Weights & Biases (wandb) and MLflow.

Who it’s for

It is designed for AI researchers and developers who need to rigorously evaluate LLMs, compare different model versions, and implement quality gates to decide if a model is ready for deployment.

Highlights

  • Extensive Benchmark Library: Built-in support for MMLU, GSM8K, HumanEval, and specialized agentic benchmarks like PinchBench.
  • Pluggable Interceptors: Ability to modify LLM traffic on the fly for caching, logging, and system message injection.
  • Flexible Execution: Supports local Docker containers and large-scale SLURM clusters for resource-intensive evaluations.
  • Comparison Tools: Dedicated CLI commands (nel compare and nel gate) for statistical comparison of runs and policy-based quality decisions.

Related

  • Project
    NVIDIA/NeMo-Agent-Toolkit

    An enterprise-grade toolkit for building, observing, and optimizing AI agents across various frameworks, enhancing their speed and accuracy through performance primitives and instrumentation.

  • Project
    modelscope/evalscope

    EvalScope is a Python‑based, all‑in‑one evaluation framework for LLMs, vision‑language, embedding, and agent models. It runs dozens of benchmarks, provides stress‑testing (latency, throughput), supports multi‑turn agent loops with tool calling, and visualises results in an interactive web dashboard. Install with `pip install evalscope` and evaluate a model via a single CLI command or a few lines of Python.

  • Project
    UKGovernmentBEIS/inspect_ai

    A framework created by the UK AI Security Institute for evaluating large language models, featuring over 200 pre-built evaluations and support for model-graded scoring.

  • Project
    NVIDIA-NeMo/Gym

    A library for evaluating and improving AI models and agents using stateful environments, providing infrastructure for scalable evaluation and RL training.

  • Project
    vllm-project/guidellm

    An SLO-aware benchmarking and evaluation platform that simulates real-world workloads to optimize LLM inference performance, resource needs, and operational limits.