MiaAI-Lab/Qwen3.8-Flash-Next-Single-DGX-Spark-TensorFold

Qwen3.8 Flash Next on one DGX Spark (TensorFold)

What it solves

This project provides a streamlined way to run the Qwen3.8 Flash Next large language model on a single NVIDIA DGX Spark system. It solves the complexity of deploying high-performance, quantized models by providing an automated setup that serves the model via an OpenAI-compatible API, supporting both text and vision inputs.

How it works

It uses the TensorFold Zig-based engine running inside an NVIDIA PyTorch container to manage model inference. The system supports two primary quantized checkpoints: a high-performance NVIDIA FP4-quantized version (the default) and an optional INT4-AutoRound version. The engine handles memory management, context windowing (up to 262,144 tokens), and parallel request processing, while providing tools to monitor performance and health.

Who it’s for

This is designed for users with access to an NVIDIA DGX Spark (or similar GB10 hardware with 128 GB of unified memory) who need to run large-scale, vision-capable AI models locally with high efficiency and low latency.

Highlights

  • Supports massive 262,144-token context windows.
  • Includes native vision support for image and video input.
  • Offers an OpenAI-compatible API for easy integration with existing tools.
  • Features automated setup and management scripts (start.sh and stop.sh).
  • Optimized for NVIDIA's quantization-aware-distilled (QAD) FP4 format for high accuracy.

Related

  • Project
    z-lab/paroquant

    ParoQuant is an INT4 quantization framework for LLMs that uses learned pairwise rotations to reduce weight outliers and maintain high accuracy during efficient inference.

  • Project
    wtdcode/vllm-backport

    A specialized fork of the vLLM inference engine optimized for running frontier models on older NVIDIA Ampere and Ada GPUs using custom quantization and memory management.

  • Project
    Project-HAMi/HAMi-core

    HAMi-core is a library that intercepts CUDA calls to enforce per‑container GPU memory and compute limits without modifying applications or drivers. It virtualizes memory, limits SM utilization via time‑sharing, and monitors usage. Build with CMake or Docker, then set `CUDA_DEVICE_MEMORY_LIMIT` and `CUDA_DEVICE_SM_LIMIT` and preload `libvgpu.so` via `LD_PRELOAD`. Logging is controlled by `LIBCUDA_LOG_LEVEL`.

  • Project
    anthropics/ClaudeForFoundationModels

    A Swift package that integrates Claude into Apple's Foundation Models framework, allowing developers to use Claude as a server-side model via native Apple APIs.