eugr/spark-vllm-docker

Docker configuration for running VLLM on dual DGX Sparks

What it solves

This project provides an optimized Docker environment and orchestration scripts specifically for running vLLM on NVIDIA DGX Spark hardware. It simplifies the complex process of setting up single-node or multi-node clusters, handling the networking requirements (InfiniBand/RDMA/NCCL) and dependency management needed for high-performance LLM inference on this specific hardware architecture.

How it works

The repository provides a suite of scripts to automate the lifecycle of a vLLM deployment:

  • Image Management: build-and-copy.sh creates Docker images (or pulls prebuilt nightly ones) and distributes them across a cluster to ensure consistency.
  • Cluster Orchestration: launch-cluster.sh manages the startup of containers across multiple nodes, automatically configuring the distributed backend (Ray or native PyTorch) and handling per-node multiprocessing arguments.
  • Model Distribution: hf-download.sh downloads models from Hugging Face and distributes them across the cluster via the ConnectX interconnect to avoid redundant internet downloads.
  • Performance Optimizations: It integrates specialized components like FlashInfer, DeepGEMM (NVIDIA branch for sm12x), and supports high-performance loading formats like fastsafetensors and InstantTensor.

Who it’s for

It is designed for AI engineers and researchers using NVIDIA DGX Spark systems who want to deploy large language models using vLLM with minimal manual configuration of distributed networking and GPU kernels.

Highlights

  • Multi-Node Support: Native support for 2-node, 3-node mesh, and larger DGX Spark clusters.
  • Hardware-Specific Tuning: Optimized for sm12x architecture with B12X kernel support.
  • Automated Distribution: Tools to sync Docker images and model weights across nodes using high-speed interconnects.
  • Flexible Build Path: Supports pulling prebuilt nightly images, building from wheels, or compiling vLLM from specific source commits/PRs.
  • Memory Management: Includes earlyoom integration to monitor host memory and prevent system crashes during heavy inference loads.

Related

  • Project
    spark-arena/sparkrun

    A CLI tool for launching and managing LLM inference workloads on NVIDIA DGX Spark systems without needing Slurm or Kubernetes.

  • Dispatch
    vLLM on the DGX Spark: Architecture, Configuration, and Local Evaluation

    vLLM provides a high-performance local inference endpoint for NVIDIA DGX Spark, enabling the deployment of large NVFP4 models like Nemotron-3-Super using a unified-memory architecture and OpenAI-compatible API.

  • Project
    docker/model-runner

    A tool for managing and deploying AI models via Docker, streamlining the process of pulling and serving LLMs using backends like llama.cpp and vLLM.

  • Project
    MiaAI-Lab/Qwen3.8-27B-SGLang-DGX-Spark

    Optimized Docker scripts for serving Qwen3.8-27B on NVIDIA DGX Spark using SGLang, featuring pre-tuned configurations for MTP, DSpark, and DFlash2 speculative decoding.

  • Project
    MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark

    A vLLM-based serving recipe for DeepSeek-V4-Flash on 2× DGX Spark, using DSpark speculative decoding, NVFP4 quantization, and a 1M-token context window.