NVIDIA/RULER

This repo contains the source code for RULER: What’s the Real Context Size of Your Long-Context Language Models?

What it solves

RULER provides a rigorous way to measure the "effective" context window of Large Language Models (LLMs). Many models claim support for very long contexts (e.g., 128k tokens), but often suffer significant performance degradation as the input length increases, even if they pass simple "needle-in-a-haystack" tests. RULER identifies where a model's performance actually begins to fail.

How it works

It generates synthetic evaluation examples across four task categories to test models with configurable sequence lengths and complexity levels:

  • Retrieval: Testing the ability to find specific information (e.g., multiple "needles" in a haystack).
  • Multi-hop Tracing: Tracking variable name-binding chains across a long text.
  • Aggregation: Extracting common words or calculating word frequencies.
  • Question Answering: Using datasets like SQuAD and HotpotQA to test comprehension over long contexts.

Who it’s for

AI researchers and developers who need to benchmark the actual long-context capabilities of LLMs beyond simple recall tests to ensure a model can handle the sequence lengths it claims.

Highlights

  • Beyond Simple Recall: Moves past basic needle-in-a-haystack tests to evaluate complex reasoning and aggregation over long sequences.
  • Configurable Complexity: Allows users to adjust task difficulty (e.g., number of hops in a chain or number of needles to find).
  • Broad Benchmarking: Includes results for a wide array of open-source and proprietary models.
  • Extensible: Provides a clear pipeline for contributors to add new synthetic tasks and evaluation metrics.

Related

  • Dispatch
    HELMET: Holistically Evaluating Long-context Language Models

    Hugging Face and Princeton researchers introduced HELMET, a comprehensive benchmark designed to replace synthetic tests like needle-in-a-haystack with diverse, controllable, and reliable real-world evaluations for long-context language models.

  • Project
    NVIDIA-NeMo/Evaluator

    An LLM evaluation framework featuring benchmark environments, a pluggable interceptor proxy for traffic management, and multi-format reporting for model performance analysis.

  • Project
    harbor-framework/harbor

    Harbor is a Python framework for running and scaling benchmarks of AI agents and language models. Installable via `pip`/`uv`, it provides a CLI (`harbor run`) that lets you specify a dataset (e.g., Terminal‑Bench‑2.0), an agent (Claude‑Code, OpenHands, etc.), and a model, then executes the benchmark locally or on cloud providers (Daytona, Modal, etc.) with configurable parallelism. It also supports creating custom benchmarks and generating RL roll‑outs.

  • Project
    lechmazur/writing

    A benchmark that evaluates LLM creative writing by having evaluator models compare short stories written to the same constrained creative briefs.

  • Project
    UKGovernmentBEIS/inspect_ai

    A framework created by the UK AI Security Institute for evaluating large language models, featuring over 200 pre-built evaluations and support for model-graded scoring.