MiaAI-Lab/Qwen3.8-27B-SGLang-DGX-Spark

Qwen3.8 27B on SGLang for DGX Spark

What this repo does

MiaAI‑Lab provides ready‑to‑run Docker scripts that let you serve the Qwen 3.8 27B large language model on an NVIDIA DGX Spark (aarch64, 128 GB unified memory) using SGLang, a high‑performance inference engine. The repository bundles:

  • Docker‑based launch scripts for three speculative decoding engines – EAGLE/MTP, DSpark, and DFlash2 – each tuned for the DGX Spark hardware.
  • A small set of benchmarking utilities (bench/) that reproduce the performance numbers reported in the README.
  • Configuration helpers (.env.sample, start‑up scripts) that let you adjust context length (up to 1 M tokens), concurrency (up to 10 simultaneous requests), quantisation mode, and CPU pinning.
  • Documentation of the exact hardware‑specific flags that gave the best measured latency/throughput on the author’s box.

In short, the repo is a deployment kit for running a 27‑billion‑parameter LLM at high speed on a specialized NVIDIA server.

Who might use it

Audience Why it matters
ML engineers / research ops who need to benchmark or deploy Qwen 3.8 27B on NVIDIA DGX Spark hardware.
AI product teams building chat‑agents, code‑assistants, or tool‑calling services that require low‑latency inference on a single powerful node.
Performance‑tuning enthusiasts who want a concrete example of how to combine SGLang’s speculative decoders (EAGLE/MTP, DSpark, DFlash2) with hardware‑specific settings (CPU‑set pinning, FP8 KV cache, memory‑fraction tuning).
Developers of downstream applications (e.g., RAG pipelines, autonomous agents) that can call the served model via the standard OpenAI‑compatible /v1/models endpoint.

How to get started (quick‑start)

  1. Clone the repo and copy the sample environment file:
    git clone https://github.com/MiaAI-Lab/Qwen3.8-27B-SGLang-DGX-Spark.git
cd Qwen3.8-27B-SGLang-DGX-Spark
cp .env.sample .env   # creates a .env with default settings
  2. Make sure Docker can see the GPUs (NVIDIA Container Toolkit installed) and you have a Hugging Face access token exported as HF_TOKEN.
  3. Pick an engine and launch it:
    # Default: DSpark (fastest on code‑type workloads)
./start-dspark.sh

# Alternative: classic EAGLE/MTP speculative decoding
./start.sh

# Alternative: DFlash2 (good for short‑chat and long‑form writing)
./start-dflash.sh
    The script pulls the appropriate SGLang image, downloads the Qwen checkpoint into ./.cache/huggingface (≈24 GB), and starts a container listening on port 8888.
  4. Test the server:
    curl http://127.0.0.1:8888/v1/models
    You should see a JSON payload confirming the model name qwen3.8-27b-sglang is ready.
  5. Stop the service when you’re done:
    ./stop.sh

What you can change (configuration)

Variable (in .env or as shell env) Default What it controls
YARN 0 Enables YaRN rope scaling for context > 262 K tokens. Must be 1 for any length above the native window.
CONTEXT_LENGTH 262144 Token window size (262 K – 1 M). Larger windows need YARN=1.
MAX_CONCURRENT_REQUESTS 10 Number of simultaneous API calls; also determines the size of the GDN state pool.
QUANT nvfp4 Chooses the checkpoint variant: nvfp4 (dense BF16 head), nvfp4-fp4 (packed FP4 head), fp8 (FP8 weights), or bf16 (full BF16). All fit in the 128 GB memory budget.
CPUSET 5-9,15-19 Pins the container to the ten high‑performance Cortex‑X5 cores on the DGX Spark, keeping the slower A725 cores free.
SPEC_STEPS / SPEC_TOPK / SPEC_DRAFT 3 / 1 / 4 Tuning knobs for the EAGLE/MTP speculative decoder. The DSpark and DFlash2 engines ignore these.
CHUNKED_PREFILL 8192 Size of pre‑fill chunks; larger values improve first‑token latency on this hardware.
DF_TARGET (DFlash2 only) nvfp4 Selects which checkpoint the DFlash2 engine loads (nvfp4, nvfp4-fp4, or bf16).
IMAGE Engine‑specific (see scripts) Allows you to point to a custom SGLang image if you have a locally built variant.

All variables can be overridden on the command line, e.g.:

MAX_CONCURRENT_REQUESTS=5 CONTEXT_LENGTH=524288 YARN=1 ./start.sh

The new values take effect after a ./stop.sh && ./start… cycle.

What the numbers look like (as measured on the author’s DGX Spark)

Engine Typical latency (seconds) – code snippet Typical latency – long essay Typical latency – short chat
DSpark (block‑7) ~51.5 s (fastest for code) ~18 s (slower than MTP) ~23 s (good)
EAGLE/MTP ~34.5 s ~24 s (best for long‑form) ~21 s
DFlash2 (NVFP4) ~50.9 s ~25.4 s 29 – 67 s (wins on streamed short‑chat once token counting is fixed)

All figures are wall‑time for the benchmark scripts in bench/ and include pre‑fill, decoding, and any speculative drafting overhead. Memory usage is ~24 GB for the dense BF16 head plus ~2.5 GB for the draft model; the packed‑FP4 variant saves a couple of gigabytes.

When to pick which engine

Goal Recommended script Reason
Agents that call tools or need code generation ./start-dspark.sh DSpark’s block‑7 draft gives the best decode speed for code‑heavy prompts.
Long‑form writing (essays, reports) ./start.sh (EAGLE/MTP) MTP’s speculative decoding shines on large, continuous outputs.
General chat or mixed workloads ./start-dflash.sh DFlash2 balances code and chat performance and beats the others on short‑chat latency after proper token counting.

Where to look for more details

  • Measured on this box – a single table in the README that lists every latency, memory‑fraction, and token‑counting caveat the author observed.
  • bench/ scripts – ndec.py, bench.sh, and ab-image.sh let you reproduce the A/B comparisons on your own hardware.
  • docs/ – contains brainstorming notes (brainstorms/*.md) and longer‑term plans (plans/*.md).

TL;DR

If you have access to an NVIDIA DGX Spark and need to run the 27 B Qwen model at low latency, this repo gives you:

  • Docker images pre‑configured for the three leading SGLang speculative decoders.
  • Scripts that automatically pull the model, apply the optimal hardware flags, and expose an OpenAI‑compatible API.
  • Simple knobs to stretch context length to 1 M tokens and run up to ten concurrent requests.

Just copy .env.sample, pick the engine you care about, run the corresponding start‑*.sh, and you’re serving Qwen 3.8 27B in minutes.

Related

  • Project
    albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4

    A set of performance optimizations for Qwen3.5-122B-A10B on NVIDIA DGX Spark, increasing inference speed from 28.3 to 52 tokens per second using hybrid quantization and speculative decoding.

  • Project
    MiaAI-Lab/DeepSeek-v4-Flash-One-DGX-Spark

    A single-node launcher for DeepSeek V4 Flash on NVIDIA DGX Spark, enabling high-throughput serving with speculative decoding and context windows up to 384k tokens.

  • Project
    eugr/spark-vllm-docker

    Docker‑based toolkit for running vLLM (including experimental B12X kernels) on single or multi‑node NVIDIA DGX Spark clusters, with scripts for building images, downloading models, and launching high‑performance inference servers.

  • Project
    MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks

    A Docker‑based, OpenAI‑compatible vLLM server for the 4‑bpw EXL3 checkpoint of GLM‑5.3‑Flash, optimized for 2‑4 NVIDIA DGX Spark boxes. It offers 850k‑token context, DFlash2 speculative decoding, and optional speed‑boost or abliteration profiles, delivering 30‑90 tokens/s decode. Configuration is via a single `.env` file; start with `./start.sh` after setting head/worker IPs.