NVIDIA Nemotron 3.5 Lightning Day-0 Support on vLLM

TL;DR

NVIDIA announced Day‑0 support for the 30‑billion‑parameter Nemotron 3.5 Lightning model on the vLLM serving engine, providing up to 4× higher throughput for always‑on agents through a hybrid mixture‑of‑experts design and three speculative decoding methods (Multi‑Token Prediction, DFlash, DSpark).

Why Nemotron 3.5 Lightning matters for agentic workloads

Nemotron 3.5 Lightning targets the "second‑tier" role in modern agent pipelines: a lightweight model that handles frequent, well‑scoped steps while a larger frontier model performs high‑level planning. Its hybrid MoE architecture activates only 3 B of the total 30 B parameters per token, reducing compute per token and enabling a 1 M‑token context window. Combined with multi‑token prediction, the model delivers up to 4× higher throughput than comparably sized open models, making it economically viable for high‑volume, always‑on agents in datacenters, cloud, or edge environments.

Core technical specifications

Feature Detail
Architecture Hybrid mixture‑of‑experts (MoE) with 30 B total parameters, 3 B active per token
Context length Up to 1 million tokens
Modalities Text input → text output
Speculative decoding Multi‑Token Prediction (MTP), DFlash, DSpark
Reasoning control Per‑request enable/disable with configurable reasoning‑token budget
Training lineage Distilled from NVIDIA Nemotron 3 Ultra; fine‑tuned on popular agent harnesses
Customization Open model; supports post‑training on domain‑specific data
Precision formats at launch BF16 and NVFP4
Supported hardware NVIDIA DGX Spark, DGX Station, RTX PRO/RTX, Jetson, H100, H200, A100, L40S, B200/GB200, B300/GB300

Getting started with vLLM

Install the vLLM container

docker pull vllm/vllm-openai:v0.27.1

docker run --rm -it \
  --gpus all \
  --ipc=host \
  --network=host \
  --entrypoint /bin/bash \
  vllm/vllm-openai:v0.27.1

Serve Nemotron 3.5 Lightning (example: 1× H100)

vllm serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 \
  --max-num-seqs 256 \
  --max-num-batched-tokens 32768 \
  --enable-prefix-caching \
  --async-scheduling \
  --mamba-backend flashinfer \
  --moe-backend humming \
  --linear-backend humming \
  --mamba-ssu-algorithm horizontal \
  --mamba-cache-mode align \
  --mamba-ssm-cache-dtype float16 \
  --enable-mamba-cache-stochastic-rounding \
  --mamba-cache-philox-rounds 5 \
  --reasoning-parser nemotron_v3 \
  --tool-call-parser qwen3_coder \
  --enable-auto-tool-choice \
  --host 0.0.0.0 \
  --port 8000

Sample OpenAI‑compatible client call

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="null")

resp = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Briefly explain: what is vLLM?"},
    ],
    temperature=1.0,
    top_p=0.95,
    max_tokens=1024,
)

choice = resp.choices[0]
print("Reasoning:", choice.message.reasoning)
print("Content:", choice.message.content)

Speculative decoding techniques

Multi‑Token Prediction (MTP)

MTP adds lightweight prediction heads that propose a small block of future tokens. The target model verifies the block, reducing the number of sequential steps.

vllm serve ... \
  --speculative_config.method mtp \
  --speculative_config.num_speculative_tokens 3

DFlash

DFlash uses a dedicated diffusion draft model to generate an entire candidate block in parallel. It requires a separate draft checkpoint.

vllm serve ... \
  --speculative_config.model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DFlash \
  --speculative_config.num_speculative_tokens 3

Draft checkpoint: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DFlash.

DSpark

DSpark blends autoregressive and diffusion‑style drafting, offering a middle ground between MTP and DFlash. It delivers the best latency‑throughput trade‑off on DGX Spark.

vllm serve ... \
  --speculative_config.method dspark \
  --speculative_config.model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark \
  --speculative_config.num_speculative_tokens 3

Draft checkpoint: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark.

Performance contributions upstream to vLLM

  • Integrated DSpark speculator into the Nemotron model definition.
  • Quantized DSpark draft head to W4A16, cutting memory use and latency.
  • Removed host‑device syncs in the draft‑and‑verify loop and enabled async scheduling.
  • Replaced default Marlin backend with Hopper‑optimized Humming backend for MoE and dense layers, yielding ~20 % throughput gain.
  • Added ReplaySSM support for Mamba2 state‑space layers, reducing recurrent‑path overhead.

Hardware‑specific deployment guides

DGX Spark (single‑user)

vllm serve ... \
  --max-num-batched-tokens 16384 \
  --compilation_config.cudagraph_capture_sizes '[1,2,4,8,16,24,32,40,48,56,64,72,80,88,96,104,112,120,128,136,144,152,160,168,176,184,192,200,208,216,224,232,240,248,256,1024,2048,4096,8192]' \
  --speculative_config.method dspark \
  --speculative_config.model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark

Pareto chart of inference performance on DGX Spark

NVIDIA H100 (single‑user)

vllm serve ... \
  --moe-backend humming \
  --linear-backend humming \
  --max-num-seqs 256 \
  --max-num-batched-tokens 32768 \
  --async-scheduling

Pareto chart of inference performance on H100 GPUs

NVIDIA Jetson (edge)

vllm serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
  --reasoning-parser nemotron_v3 \
  --kv-cache-dtype fp8 \
  --trust-remote-code \
  --max-num-batched-tokens 16384 \
  --enable-prefix-caching \
  --mamba-backend flashinfer \
  --mamba-ssm-cache-dtype float16 \
  --enable-mamba-cache-stochastic-rounding \
  --mamba-cache-philox-rounds 5 \
  --mamba-cache-mode align

Accuracy and efficiency frontier

Nemotron 3.5 Lightning inherits capabilities from Nemotron 3 Ultra and is fine‑tuned on agent‑centric datasets. Benchmarks (PinchBench) show the model achieving comparable accuracy to larger models such as Qwen 3.6 35B while completing 10 000 tasks up to 30 % faster. Efficiency frontier line chart

How to obtain the model

Comparison with Nemotron 3 Nano

Nemotron 3 Nano introduced the hybrid Mamba‑Transformer MoE design. Nemotron 3.5 Lightning extends it by:

  1. Distilling frontier‑model capabilities from Nemotron 3 Ultra.
  2. Optimizing training for agent harnesses and multi‑turn workflows.
  3. Adding three speculative decoding paths (MTP, DFlash, DSpark) to accelerate generation. The net effect is higher accuracy on agentic tasks and up to 4× throughput improvement.

Acknowledgements

NVIDIA contributors: Nirmal Kumar Juluru, Anusha Pant, Amir Klein, Faradawn Yang, Nave Assaf, Ryan Stewart, Alex Steiner, Bita Rouhani.

Sources

Related

  • Dispatch
    NVIDIA Nemotron 3 Ultra Support on vLLM

    vLLM announces Day-0 support for NVIDIA Nemotron 3 Ultra, a 550B parameter model optimized for long-running autonomous agent workflows through a hybrid Transformer-Mamba MoE architecture.

  • Dispatch
    NVIDIA Nemotron 3 Super Support in vLLM

    vLLM now supports NVIDIA Nemotron 3 Super, a 120B parameter hybrid MoE model optimized for multi-agent AI with a 1 million token context window and high inference efficiency.

  • Dispatch
    NVIDIA Nemotron 3.5 Lightning Release

    NVIDIA Nemotron 3.5 Lightning is a 30B parameter open model with 3B active parameters per token, designed for local agentic workflows and multi-step tasks with a 1M token context window.

  • Dispatch
    NVIDIA Nemotron 3 Nano Omni Support in vLLM

    vLLM now supports NVIDIA Nemotron 3 Nano Omni, a highly efficient 30B MoE multimodal model that unifies vision, audio, and language reasoning in a single loop to power agentic AI.

  • Dispatch
    NVIDIA Nemotron 3.5 Lightning and NeMo Switchyard Release

    NVIDIA has released Nemotron 3.5 Lightning, a 30B MoE model for agentic workflows, and NeMo Switchyard, an open-source routing library to optimize model ensembles.