MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold

GLM-5.3-Flash EXL3 on 2x DGX Spark with TensorFold

What it solves

Running a massive language model like GLM-5.3-Flash usually demands expensive GPUs and plenty of memory. This project shows how to serve it on two modest NVIDIA DGX Sparks (each with 128 GB unified memory) connected by a fast link, using TensorFold to split the model across both machines and applying aggressive quantization (4-bit dense weights, FP8 KV cache) to fit the model and support the full 1,048,576-token context window. It also solves the problem of concurrency: up to four requests can share a common key-value cache pool, and it includes many patches that fix issues, add features (like the Anthropic Messages API), and improve throughput.

How it works

The recipe uses two DGX Sparks as a head and a worker, linked via a ConnectX-7 QSFP cable with RoCE networking. Both machines run TensorFold v0.6.0 in NVIDIA's PyTorch container, one rank per machine, with a set of patches that enhance functionality and stability. The model itself is downloaded as a custom EXL3 quantized checkpoint (4-bit dense weights, BF16 elsewhere) plus an optional DFlash2 drafter for speculative decoding. A single ./start.sh script handles setup, verification, launching, and a smoke test, and it provides an OpenAI-compatible API endpoint on port 8888. The system uses an FP8 KV cache pool to serve multiple requests concurrently, supports image and video input by running the model's vision tower on the head, and includes tool calling, structured outputs (via xgrammar), and prompt reuse to speed up repeated system prompts.

Who it's for

Developers and researchers who want to deploy GLM-5.3-Flash on relatively affordable dual-GB10 hardware, need long context (up to 1M tokens), and want a drop-in OpenAI-compatible endpoint for building applications that use text, images, and videos. It requires some comfort with Docker, SSH, and NVIDIA hardware setup.

Highlights

  • Serves GLM-5.3-Flash with up to 4 concurrent requests over a shared FP8 KV pool (~2.9M tokens total).
  • Supports the full 1,048,576-token context window per request.
  • Decode throughput up to ~108.8 tok/s aggregate and ~1,950 tok/s prefill; time to first token as low as 170 ms.
  • Prompt reuse: an identical 64k-token prompt second visit takes under 0.07 s instead of 34 s.
  • Handles images (JPEG/PNG/WebP) and videos (MP4/WebM/MOV/MKV) via OpenAI-style content parts.
  • Includes tool calling, structured outputs, and an Anthropic Messages API compatibility patch.
  • Provides a one-command setup (./start.sh) that prepares both Sparks and launches the server, with logs and health checks.
  • Quality benchmarks on GSM8K (98.8%), HumanEval (95.7%), and HumanEval+ / MBPP+ (86.5%) with the 4-bit / FP8 configuration.

Related

  • Project
    zai-org/GLM-5

    GLM‑5 is an open‑weight LLM family (up to 744 B parameters, 1 M‑token context) focused on coding, cyber‑security, and long‑horizon agentic tasks. It introduces sparse‑plus‑linear attention (Flash), the IndexShare efficiency trick, and a reasoning‑effort API. Models are downloadable from Hugging Face/ModelScope and can be served with SGLang, vLLM, Transformers, KTransformers, or Ascend NPU. Fine‑tuning is supported via the slime RL framework and ms‑swift. The series claims state‑of‑the‑art open‑source performance on several benchmarks.

  • Project
    MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks

    A deployment toolkit for running the GLM-5.3-Flash LLM on NVIDIA DGX Spark clusters, featuring automated orchestration, speculative decoding, and an OpenAI-compatible API.

  • Project
    NVIDIA/TensorRT-Edge-LLM

    TensorRT Edge‑LLM is NVIDIA’s C++/Python inference runtime for running LLMs and multimodal models on Jetson, DRIVE and DGX Spark edge hardware. It converts Hugging Face checkpoints to TensorRT engines (via ONNX or a direct builder), offers quantization, in‑flight batching, speculative decoding, and a vLLM‑style Python API with an OpenAI‑compatible server. The library targets automotive, robotics, industrial IoT and consumer edge use‑cases, delivering high‑throughput, low‑latency inference on resource‑constrained GPUs.

  • Dispatch
    GLM-5.3-Flash Inference Infrastructure: How an AI Agent Built a 100k‑Accelerator Service

    GLM-5.3-Flash’s production inference stack was built in under two weeks by an Infra Agent powered by GLM‑5.3, achieving a three‑fold throughput boost on a 100,000‑accelerator Chinese AI chip cluster through dense feedback loops and aggressive memory optimizations.

  • Dispatch
    vLLM on the DGX Spark: Architecture, Configuration, and Local Evaluation

    vLLM provides a high-performance local inference endpoint for NVIDIA DGX Spark, enabling the deployment of large NVFP4 models like Nemotron-3-Super using a unified-memory architecture and OpenAI-compatible API.