vLLM on the DGX Spark: Architecture, Configuration, and Local Evaluation

vLLM enables fast, efficient local inference on the NVIDIA DGX Spark, bridging the gap between laptop-scale development and data center GPU serving. By pairing an OpenAI-compatible API with advanced memory, batching, and telemetry controls, vLLM allows developers to run large NVFP4 models locally on the DGX Spark's unique hardware architecture.

DGX Spark Architecture and Memory Model

The NVIDIA DGX Spark is built around the GB10 Grace Blackwell SoC, which utilizes a unified CPU and GPU memory pool. This architecture significantly impacts how inference workloads are configured and executed.

Unified Memory and Model Capacity

The shared CPU/GPU memory pool allows developers to load larger NVFP4 models than would be possible with a fixed dedicated GPU-memory pool. This makes it practical to run models with up to 200 billion parameters on a single Spark, depending on the model architecture and runtime configuration. vLLM manages this unified pool using flags such as --gpu-memory-utilization, --max-model-len, and --max-num-seqs, alongside a paged KV cache to balance model size, context length, and concurrency.

Hardware-Specific Validation

Deployments on DGX Spark must use vLLM builds, container image tags, and runtime settings validated for the sm_121 target. Configurations adapted from larger GPU systems should be treated as engineering checklists for kernel support rather than performance benchmarks.

NVFP4 MoE Optimization

DGX Spark is specifically well-suited for NVFP4 Mixture-of-Experts (MoE) serving. NVFP4 reduces memory pressure and improves prefill behavior. MoE models with approximately 10-15 billion active parameters are a strong fit because the smaller active parameter set optimizes decode performance on the system's memory bandwidth.

vLLM Capabilities for Local Serving

vLLM provides several key features that optimize the DGX Spark's local, small-batch inference profile.

Paged KV Cache and Continuous Batching

To avoid the inefficiencies of static batching in chat workloads, vLLM uses continuous batching to admit and evict requests at every decode step. When paired with a paged KV cache, the DGX Spark can support multiple in-flight requests without excessive memory fragmentation. In tests with a 120B NVFP4 MoE model, KV-cache utilization typically remained below 5% for single users and below 30% for small-batch demo traffic.

OpenAI-Compatible Streaming

The vLLM endpoint on Spark uses an OpenAI-compatible API (e.g., http://localhost:8000/v1), allowing existing client code to be reused. The use of stream=true is critical for local responsiveness, rendering tokens as they arrive to minimize perceived latency in chat, coding, and agentic workflows.

Observability via Prometheus

vLLM's Prometheus endpoint (/metrics) allows developers to monitor the health of the local appliance. Key signals for DGX Spark include:

• KV-cache utilization (vllm:kv_cache_usage_perc)
• Prompt and generation token counters
• TTFT (Time to First Token) and inter-token-latency histograms

Runtime Configuration and Deployment

Successful deployment on DGX Spark requires matching the runtime flags to the system's unified-memory profile and the specific model recipe.

Model Selection Guidance

Model choice is the primary performance lever. 100-130B MoE NVFP4 models with 10-15B active parameters are the most efficient fit for the Spark's memory capacity and decode rate.

Critical vllm serve Flags

• --gpu-memory-utilization: Must be tuned to leave headroom in the unified memory pool for the OS, container runtime, and KV cache growth.
• --max-model-len 131072: Sets the maximum prompt and completion length. vLLM schedules based on active context rather than reserving the full length for every request.
• --max-num-seqs 4: Keeps concurrent decode streams low to prevent per-token bandwidth taxes from spiking TTFT.
• Automatic Prefix Caching: Enabled by default in vLLM V1, this reuses KV blocks across requests sharing an opening prompt, which is highly beneficial for long system prompts.

JIT Pre-warming and Loading

Cold-start latency is a significant factor; the first request after boot can trigger Inductor and FlashInfer JIT codegen, taking roughly 25 seconds for Nemotron-3-Super. Developers should send a small "ping" request at startup to warm the kernels. Additionally, while initial weight loading can take 10-15 minutes, paths like fastsafetensors or InstantTensor can be evaluated to reduce this time.

Local Evaluation Results: Nemotron-3-Super

Evaluation of the Nemotron-3-Super-120B-A12B-NVFP4 model on a single DGX Spark showed consistent decode throughput in the 22.7–23.7 tok/s range across various scenarios.

Scenario	Prompt tok	Gen tok	TTFT	Total latency	Prefill tok/s	Decode tok/s
Typical judge call	58	2	0.42 s	~0.53 s	140	~23
Medium prompt, short gen	1,834	32	1.12 s	1.12 s	1,636	23.7
Long prompt, short gen	7,234	32	3.85 s	~5.26 s	1,877	22.7
Medium prompt, long gen	1,834	108	1.12 s	~5.74 s	1,639	23.4
Long prompt, long gen	7,234	124	3.84 s	~9.26 s	1,884	22.9

Key Performance Insights

• Prefill Scaling: Prefill throughput increases as prompts get larger, climbing from 140 to nearly 1,900 tokens per second, as the overhead is amortized across more tokens.
• Decode Stability: Decode throughput remains stable regardless of prompt length, dictated by active parameter count and the FP4 kernel path.
• TTFT: Time to First Token roughly triples when the prompt size increases fourfold.

Operational Summary

To optimize vLLM on DGX Spark, developers should prioritize 100-130B MoE NVFP4 models, use official vLLM images validated for sm_121, and tune --gpu-memory-utilization for the unified memory pool. Pre-warming the JIT and utilizing the /metrics endpoint ensures a predictable, interactive local inference experience.