eugr/spark-vllm-docker

What it is

spark‑vllm‑docker is a set of Dockerfiles, helper scripts, and ready‑made recipes that let you run the vLLM inference engine on NVIDIA DGX Spark clusters (or a single DGX Spark node). It automates building a Docker image that contains vLLM (and optional experimental B12X kernels), pulls model weights from Hugging Face, and launches the vLLM server across one or many Spark nodes using either Ray or vLLM’s native PyTorch‑distributed backend.

Who it’s for

• Engineers who have access to DGX Spark hardware and want to serve large LLMs (e.g., Qwen‑3.8, DeepSeek‑V4, GLM‑5.x, Nemotron 3.5) at high throughput.
• Teams that need multi‑node inference with InfiniBand/RDMA, KV‑cache FP8, speculative decoding, and fast model loading via InstantTensor or the experimental B12X loader.
• Users who prefer a reproducible Docker‑based workflow rather than building vLLM manually on each node.

Core features (as described in the README)

Feature	Details
Docker image	eugr/spark-vllm:latest (regular) or eugr/spark-vllm-b12x:latest (experimental B12X kernels). Images are built from nightly vLLM wheels or from source with optional --use-wheels, --rebuild-vllm, --rebuild-flashinfer flags.
Multi‑node support	Launch scripts (launch‑cluster.sh, run‑recipe.sh) discover Spark nodes, set up password‑less SSH, and start vLLM workers with NCCL over InfiniBand/RoCE.
Model loading	Supports standard safetensors, InstantTensor zero‑copy, and the B12X loader for even faster, lower‑memory loads.
Speculative decoding	Built‑in configs for DFlash, MTP, DSpark, etc., allowing up to 8 speculative tokens per step.
Memory tricks	FP8 KV‑cache, VLLM_WSL2_ENABLE_PIN_MEMORY, early‑oom monitor, CPU heap trimming, and optional PLE‑table offload to disk.
Convenient recipes	YAML files for popular models (Qwen‑3.8‑27B, DeepSeek‑V4‑Flash‑Vision, GLM‑5.3, Nemotron‑3.5, etc.) that pre‑configure model IDs, parallelism, tokenizer, and performance flags.
Extensible	--apply‑vllm‑pr can pull upstream vLLM pull‑request diffs at launch time; mods/ directory holds custom loader patches.
Health check	Simple curl http://localhost:8000/health and OpenAI‑compatible /v1/models endpoints.

How to get started (quick‑start summary)

1. Clone the repo on the head Spark node.
2. Build & copy the Docker image: ./build-and-copy.sh (add --exp-b12x for the experimental image).
3. Download the model weights: ./hf-download.sh <model‑repo>.
4. Launch the server:

   ./launch-cluster.sh --solo -t vllm-node \
     exec vllm serve <model> \
       --host 0.0.0.0 --port 8000 \
       --kv-cache-dtype fp8 ...
   

   (Replace vllm-node with vllm-node-b12x and add the many extra flags shown in the README for B12X‑based recipes.)
5. Test the endpoint with curl and point your client to http://<head‑node-ip>:8000/v1.

Why it matters

Running vLLM on a single GPU is straightforward, but scaling inference to multiple DGX Spark nodes while keeping latency low requires careful networking, memory management, and kernel selection. This repository bundles all those knobs into reproducible Docker images and scripts, letting practitioners focus on the model rather than the low‑level cluster plumbing.

---

All information above is taken directly from the repository’s README; no additional features have been inferred.