NVIDIA Nemotron 3 Super Support in vLLM

vLLM now supports NVIDIA Nemotron 3 Super, an open hybrid Mixture-of-Experts (MoE) model designed specifically for complex multi-agent AI applications. This integration allows developers to deploy a model that balances high intelligence with the compute efficiency required for large-scale agentic systems.

High-Efficiency Architecture for Multi-Agent AI

Nemotron 3 Super utilizes a hybrid Transformer-Mamba architecture and a Mixture-of-Experts (MoE) design to minimize the "thinking tax" and context overhead associated with agentic workflows.

Key Technical Specifications

  • Model Parameters: 120 billion total parameters, with only 12 billion active parameters during inference.
  • Context Window: Supports up to 1 million tokens, reducing goal drift and solving the "context explosion" problem caused by repeated history and tool outputs in multi-agent systems.
  • Throughput: Provides up to 4x higher throughput than conventional massive models and up to 5x higher throughput compared to the previous Nemotron Super model.
  • Multi-Token Prediction (MTP): Accelerates long-form text generation by predicting several future tokens simultaneously in a single forward pass.
  • Latent MoE: Enables the activation of four experts for the inference cost of a single expert.
  • Thinking Budget: Allows for optimal accuracy by controlling the amount of reasoning token generation.

Performance and Openness

Nemotron 3 Super is positioned as a leading open model on the Artificial Analysis Intelligence and Openness indices. It is fully open, providing open weights, datasets, and recipes to allow for customization and secure deployment on private infrastructure.

According to Artificial Analysis benchmarks, the model achieves leading accuracy in its size category—up to 2x higher accuracy than the previous Nemotron Super model—while maintaining high efficiency compared to other open models of similar size.

Deployment with vLLM

vLLM enables optimized inference for Nemotron 3 Super across various precision formats, including BF16, FP8, and NVFP4. On Blackwell GPUs, NVFP4 delivers 4x higher throughput than FP8 on H100 while maintaining accuracy.

Hardware Support

Supported GPUs include:

  • NVIDIA B200
  • NVIDIA H100
  • DGX Spark
  • RTX 6000

Quick Start Implementation

To serve the model using vLLM (version 0.17.1), a 4x H100 setup can be configured via an OpenAI-compatible API with the following command:

# BF16
vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \
    --kv-cache-dtype fp8 \
    --tensor-parallel-size 4 \
    --trust-remote-code \
    --served-model-name nemotron \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --reasoning-parser nemotron_v3

Once the server is active, the model can be prompted using the OpenAI Python client, accessing both reasoning_content and the final content in the response.

Sources