NVIDIA Nemotron 3 Ultra Support on vLLM

vLLM has announced Day-0 support for NVIDIA Nemotron 3 Ultra, a frontier-class reasoning model specifically designed for long-running autonomous agent workflows. This integration allows developers to deploy high-throughput, agentic AI systems capable of complex orchestration, coding, and deep research using an OpenAI-compatible API.

Nemotron 3 Ultra Technical Specifications

Nemotron 3 Ultra is an open-weights model designed to maintain reasoning depth while ensuring inference speed for practical deployment in persistent agentic systems.

• Architecture: A Mixture of Experts (MoE) utilizing a Hybrid Transformer-Mamba architecture.
• Parameter Count: 550B total parameters, with 55B active parameters.
• Context Window: Supports up to 1M tokens.
• Modalities: Text input and text output.
• Precision Support: High-throughput inference via NVFP4 and BF16.
• Hardware Compatibility:
  • BF16: Supported on 8x GB200/B200/GB300/B300, 16x H100, or 8x H200 GPUs.
  • NVFP4: Supported on 4x GB200/B200/GB300/B300 or 8x H100 GPUs (NVFP4 checkpoints specifically work on Blackwell GPUs).

Architectural Innovations for Agentic Reasoning

To balance efficiency and accuracy in high-capacity reasoning, Nemotron 3 Ultra implements several architectural advancements:

Hybrid Mamba-Transformer Layers

Combining Mamba layers for sequence efficiency in long-context workloads with Transformer layers to ensure precise recall of specific facts from large context windows.

Latent MoE and Multi-Token Prediction (MTP)

Latent MoE enables more efficient expert routing across diverse tasks like code generation and tool calls. Multi-Token Prediction (MTP) reduces generation time by predicting multiple future tokens in a single forward pass, increasing throughput for multi-turn workflows.

Agent-Specific Post-Training

The model is post-trained using NVIDIA NeMo RL and Gym across various agent harnesses. This optimization focuses on multi-turn workflows where agents must plan, call tools, read observations, delegate to sub-agents, and recover from errors, rather than simple single-turn chat.

Performance and Cost Efficiency

Nemotron 3 Ultra is positioned as a leader among open models for agent productivity, instruction following, and long-context tasks. According to the source material, the model provides leading throughput and reduces costs by up to 30% compared to other leading open models.

Deployment with vLLM

vLLM served as a critical component of the Nemotron 3 Ultra training workflow, providing high-throughput multi-node inference for rollouts and model evaluation. It currently functions as the generation backend for reinforcement learning rollouts within NeMo RL, integrating with NeMo Gym for multi-step training environments.

Serving the Model

Developers can serve Nemotron 3 Ultra through vLLM using an OpenAI-compatible API. A typical configuration for an 8x B200 setup includes the following flags:

• --tensor-parallel-size 8
• --kv-cache-dtype fp8
• --speculative_config.method mtp (utilizing Multi-Token Prediction)
• --reasoning-parser nemotron_v3
• --tool-call-parser qwen3_coder

Model weights are available on Hugging Face in both BF16 and NVFP4 formats.