NVIDIA Nemotron 3 Nano open evaluation recipe with NeMo Evaluator

TL;DR

NVIDIA released the 30‑billion‑parameter Nemotron 3 Nano A3B model together with a fully open evaluation recipe built on the NeMo Evaluator library, enabling anyone to reproduce its benchmark scores and audit every step of the evaluation pipeline.

Why open evaluation matters

Open evaluation eliminates hidden variables that can inflate or deflate reported scores. By publishing the exact configuration files, inference settings, logs, and artifact layout, NVIDIA makes it possible to verify whether a model’s performance reflects genuine capability or merely a tuned benchmark setup.

A single, consistent evaluation system

NeMo Evaluator provides a unified interface for defining benchmarks, prompts, runtime parameters, and inference back‑ends. The same YAML configuration can be reused across model releases, preventing silent changes that would otherwise break longitudinal comparisons.

Methodology independent of inference setup

NeMo Evaluator separates the evaluation workflow from the inference engine. Whether the model is accessed via NVIDIA’s hosted endpoint, Hugging Face Inference API, OpenRouter, or a local deployment, the same evaluation definition applies, ensuring comparable results across infrastructure changes.

Scaling beyond one‑off experiments

The launcher, artifact layout, and configuration model in NeMo Evaluator are designed for repeated, large‑scale runs. Teams can move from a single benchmark validation to full model‑card suites without rewriting scripts, preserving methodological consistency over time.

Auditability with structured artifacts and logs

Each run automatically generates:

  • Per‑task results.json files containing raw scores.
  • Comprehensive execution logs for debugging.
  • A deterministic directory hierarchy (artifacts/ and logs/).

These artifacts make it trivial to trace how a final number was derived and to perform deeper analyses of model behavior.

A shared evaluation standard

By releasing Nemotron 3 Nano with its complete evaluation recipe (see the GitHub link in the model card), NVIDIA offers the community a reference methodology. Using the same configuration and tools aligns benchmark selection, execution, and interpretation across different models and providers.

Open‑source tooling: NeMo Evaluator

NeMo Evaluator orchestrates hundreds of benchmarks from multiple harnesses, including:

  • NeMo Skills – instruction‑following, tool use, and agentic tasks.
  • LM Evaluation Harness – base‑model and pre‑training benchmarks.

Each harness retains its native logic and datasets, while NeMo Evaluator standardizes configuration, execution, and logging. This eliminates the need for bespoke scripts and yields consistent, auditable results.

Open configurations

The exact YAML used for the Nemotron 3 Nano model card is publicly available. It specifies:

  • Model inference and deployment settings.
  • Benchmark and task selection.
  • Sampling, repeat, and prompt‑template parameters.
  • Runtime controls (parallelism, timeouts, retries).
  • Output paths and artifact layout.

Running the same YAML guarantees identical evaluation methodology.

Open logs and artifacts

After execution, the output directory contains:

results_nvidia_nemotron_3_nano_30b_a3b/
├── artifacts/
│   └── <task_name>/results.json
└── logs/
    └── stdout.log

These files let users inspect scoring logic, debug unexpected outcomes, and compare results across runs.

Reproducibility workflow

To reproduce Nemotron 3 Nano’s scores:

  1. Obtain the model checkpoint or hosted endpoint.
  2. Pull the published NeMo Evaluator config from the GitHub repository.
  3. Execute a single CLI command (nemo-evaluator-launcher run …).
  4. Inspect the generated logs and results.json files and compare them to the model card.

The same workflow works for any model, provided the endpoint is reachable and the configuration is adapted accordingly.

Running the evaluation suite

A typical run looks like:

pip install nemo-evaluator-launcher
export NGC_API_KEY="your-ngc-api-key"
export HF_TOKEN="your-huggingface-token"
export JUDGE_API_KEY="your-judge-api-key"  # only for judge‑based benchmarks

nemo-evaluator-launcher run \
  --config /path/to/examples/nemotron/local_nvidia_nemotron_3_nano_30b_a3b.yaml

Options such as --dry-run or limit_samples allow previewing the job or testing with a reduced dataset.

Selecting individual benchmarks

Use the -t flag to run specific tasks, e.g.:

# Only MMLU‑Pro
nemo-evaluator-launcher run --config … -t ns_mmlu_pro

# Only coding benchmarks
nemo-evaluator-launcher run --config … -t ns_livecodebench

Monitoring and inspecting results

nemo-evaluator-launcher status            # job status
nemo-evaluator-launcher logs <job-id>    # stream logs

The structured results.json files enable direct comparison with the scores reported in the Nemotron 3 Nano model card.

Interpreting minor score variations

Small differences between reproduced scores and the published numbers are expected due to the stochastic nature of LLM inference (sampling, parallel execution, judge variability, etc.). The goal of open evaluation is methodological consistency, not bit‑wise identical outputs. To ensure a valid reproduction, verify:

  • The YAML configuration matches the published version.
  • The same benchmark versions and prompt templates are used.
  • The intended model endpoint and chat template are selected.
  • Runtime parameters (repeats, parallelism, timeouts) are unchanged.
  • All artifacts and logs are present and correctly structured.

When these conditions hold, the reproduced results are a faithful representation of the reference methodology.

Implications for the AI community

The Nemotron 3 Nano open evaluation package demonstrates a practical path toward transparent, reproducible benchmarking for large language models. By exposing every component of the evaluation pipeline, NVIDIA enables:

  • Independent verification of claimed performance.
  • Fair, apples‑to‑apples model comparisons across providers.
  • Scalable, automated evaluation pipelines for research and production.
  • Community contributions to extend the benchmark catalog.

This approach shifts the focus from isolated scores to the trustworthiness of the evaluation process itself.

Getting started yourself

  1. Clone the NeMo Evaluator repository.
  2. Follow the step‑by‑step tutorial in the nano-v3-reproducibility.md example.
  3. Point the configuration at any model you wish to evaluate (hosted or self‑served).
  4. Run the suite, inspect the structured outputs, and share your findings.

Conclusion

NVIDIA’s release of Nemotron 3 Nano with a complete, open‑source evaluation recipe marks a significant step toward a shared, auditable standard for LLM benchmarking. The NeMo Evaluator library makes it possible to run heterogeneous benchmarks consistently, regardless of inference backend, and to keep a full provenance trail of configurations, logs, and results. This transparency empowers researchers and developers to validate claims, compare models reliably, and build larger‑scale evaluation pipelines without reinventing the wheel.

Join the community – contribute new benchmarks, report issues, or propose enhancements on the NeMo Evaluator GitHub repository.

Sources