NVIDIA Nemotron 3 Nano Omni release notes / what's new

NVIDIA has introduced Nemotron 3 Nano Omni, an omni-modal understanding model designed for real-world document analysis, automatic speech recognition (ASR), long audio-video understanding, agentic computer use, and general multimodal reasoning. The model extends the Nemotron multimodal line by integrating text, image, video, and audio capabilities into a single system capable of processing very long multimodal contexts.

Key Capabilities and Use Cases

Nemotron 3 Nano Omni is optimized for five primary classes of workloads:

• Real-world Document Analysis: The model handles 100+ page documents, focusing on layout, tables, figures, formulas, and cross-page references for complex materials like contracts and technical papers.
• Automatic Speech Recognition (ASR): It provides high-quality transcription across diverse audio conditions, including long-form audio with varying accents and background noise.
• Long Audio-Video Understanding: The model jointly reasons over mixed audio and visual evidence, such as screen recordings with narration or meeting archives.
• Agentic Computer Use: Specifically trained for GUI environments, the model can interpret screenshots, monitor UI state, and assist with action selection and workflow automation.
• General Multimodal Reasoning: It synthesizes information across long context windows and multiple modalities to perform multi-step reasoning and calculations.

Technical Architecture

Nemotron 3 Nano Omni uses a unified encoder-projector-decoder design. The language backbone is Nemotron 3 Nano 30B-A3B, which is paired with the C-RADIOv4-H vision encoder and the Parakeet-TDT-0.6B-v2 audio encoder. These encoders connect to the LLM backbone via lightweight 2-layer MLP projectors.

Hybrid Mamba-Transformer-MoE Backbone

To maintain reasoning performance while scaling to long multimodal contexts, the backbone interleaves three components:

• 23 Mamba selective state-space layers for efficient long-context processing.
• 23 MoE layers (128 experts, top-6 routing, and a shared expert) for conditional capacity.
• 6 grouped-query attention layers to preserve global interaction and expressivity.

Vision and Video Innovations

• Dynamic Resolution: The model replaces tiling with dynamic resolution processing at native aspect ratios, using between 1,024 and 13,312 visual patches per image. This is critical for OCR-heavy documents and GUI layouts.
• Conv3D Temporal Compression: For video, a dedicated Conv3D tubelet embedding path fuses pairs of consecutive frames into a single tubelet, halving the vision tokens the language model must attend to.
• Efficient Video Sampling (EVS): This inference-time feature drops redundant static video tokens while keeping dynamic tokens where the video is changing, further reducing latency and improving throughput.

Native Audio Integration

Powered by Parakeet-TDT-0.6B-v2, the model processes audio sampled at 16 kHz. It is trained with inputs up to 1,200 seconds (20 minutes), while the LLM max context length supports over 5 hours of audio. Audio, visual, and text tokens are interleaved and processed jointly within a shared multimodal sequence.

Performance and Benchmarks

Nemotron 3 Nano Omni demonstrates substantial gains over Nemotron Nano V2 VL and outperforms Qwen3-Omni 30B-A3B in several domains:

Task	Benchmark	Nemotron 3 Nano Omni	Nemotron Nano V2 VL	Qwen3-Omni 30B-A3B
Document understanding	OCRBenchV2-En	65.8	61.2	-
	MMLongBench-Doc	57.5	38.0	49.5
	CharXiv reasoning	63.6	41.3	61.1
GUI	ScreenSpot-Pro	57.8	5.5	59.7
	OSWorld	47.4	11.0	29.0
Video understanding	Video-MME	72.2	63.0	70.5
Video + Audio understanding	WorldSense	55.4	-	54.0
	DailyOmni	74.1	-	73.6
Voice interaction	VoiceBench	89.4	-	88.8
ASR	HF Open ASR	5.95	-	6.55

In terms of efficiency, Nemotron 3 Nano Omni delivers 7.4x higher system efficiency for multi-document use cases and 9.2x higher system efficiency for video use cases compared to other open omni models with similar interactivity.

Training and Data Infrastructure

Training was conducted on NVIDIA H100 nodes (scaling from 32 to 128) using Megatron-LM, Transformer Engine, and Megatron Energon. Post-SFT reinforcement learning was handled by NeMo-RL and NeMo Gym across B200 and H100 clusters.

Reinforcement Learning (RL)

• Text RL: Evaluates the model's ability to perform tool calling, code writing, and multi-part planning.
• Omni RL: Trains the model to reason across images, video, audio, and text. A verifier suite includes unanswerable cases to teach the model to abstain from answering when evidence is insufficient to prevent hallucinations.

Synthetic Data Generation

NVIDIA used NeMo Data Designer to generate approximately 11.4M synthetic QA pairs (~45B tokens) from real-world PDFs to strengthen long-context document reasoning, resulting in a 2.19x improvement in MMLongBench-Doc accuracy.

Availability

Checkpoints are available on Hugging Face in BF16, FP8, and NVFP4 formats. The technical report is available on arXiv (2604.24954).