NVIDIA Nemotron 3 Nano Omni Support in vLLM

NVIDIA Nemotron 3 Nano Omni Support in vLLM

vLLM now supports NVIDIA Nemotron 3 Nano Omni, an open multimodal model designed to power agentic AI by unifying vision, audio, and language reasoning into a single efficient loop. This integration allows developers to deploy a model that replaces fragmented multimodal stacks—where separate models for vision, speech, and language are typically bolted together—thereby reducing inference hops, orchestration overhead, and context fragmentation.

Unified Multimodal Architecture

Nemotron 3 Nano Omni collapses separate perception models into a single reasoning loop capable of processing text, images, video, and audio inputs simultaneously. This architecture is specifically designed for enterprise agent workflows that require continuous perception of screens, documents, and audio-visual streams.

Technical Specifications

  • Architecture: Mixture of Experts (MoE) with a Hybrid Transformer-Mamba Architecture.
  • Model Size: 30B total parameters, with only 3B active parameters per forward pass.
  • Context Length: 256K tokens.
  • Temporal Processing: Uses 3D convolution layers (Conv3D) to handle temporal-spatial data in video efficiently.
  • Modalities: Supports text, image, video, and audio as inputs, with text as the output.

Performance and Efficiency Gains

Nemotron 3 Nano Omni delivers significantly higher throughput and lower compute costs compared to other open omni models and previous iterations.

Throughput and Scalability

The model achieves 9x higher throughput than other open omni models with the same level of interactivity. Specifically, when measured at a fixed per-user interactivity threshold, the model demonstrates:

  • Video Use Cases: 9.2x higher throughput.
  • Multi-Document Use Cases: 7.4x higher throughput.

Computational Efficiency

To maintain high performance without prohibitive costs, the model employs several optimization techniques:

  • Efficient Video Sampling (EVS): Enables the processing of longer videos within the same compute budget via temporal-aware perception.
  • Quantization Support: Supports BF16, FP8, and NVFP4 precision to accelerate inference and increase request capacity on the same GPU.
  • Hardware Compatibility: Optimized for NVIDIA B200, H100, H200, A100, L40S, DGX Spark, and RTX 6000 GPUs.

Multimodal Intelligence and Accuracy

Nemotron 3 Nano Omni shows marked improvements in multimodal reasoning accuracy over the previous Nemotron Nano VL V2 model, leading to top placements across six industry leaderboards.

Benchmark Success

The model leads in several specialized categories:

  • Document Intelligence: Top performance on MMlongbench-Doc and OCRBenchV2.
  • Video and Audio Understanding: Leading results on WorldSense, DailyOmni, and VoiceBench.
  • ### MediaPerf: Achieved the highest throughput across every task and the lowest inference cost for video-level tagging.

Training Methodology

Accuracy gains were driven by multi-environment reinforcement learning using NVIDIA NeMo RL and NeMo Gym across text, image, audio, and video environments, which improved instruction following and the convergence of multimodal answers.

Deployment with vLLM

Nemotron 3 Nano Omni can be served via an OpenAI-compatible API using vLLM. The deployment requires vllm[audio]==0.20.0 and supports advanced configurations such as video pruning and auto-tool choice.

Example Server Command

python3 -m vllm.entrypoints.openai.api_server \
    --model "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16" \
    --served-model-name nemotron \
    --trust-remote-code \
    --dtype auto \
    --host 0.0.0.0 \
    --port 5000 \
    --tensor-parallel-size 1 \
    --max-model-len 131072 \
    --media-io-kwargs '{"video":{"num_frames":512,"fps":1}}' \
    --video-pruning-rate 0.5 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --reasoning-parser nemotron_v3

Sources