spark-arena/sparkrun

sparkrun - launch, manage, and stop LLM inference workloads on NVIDIA DGX Spark systems. Live support: https://discord.com/invite/GH5kRgv6ZD

What it solves

Sparkrun simplifies the deployment and management of LLM inference workloads on NVIDIA DGX Spark systems. It removes the need for complex orchestration tools like Slurm or Kubernetes, providing a streamlined command-line interface for launching and managing models.

How it works

Sparkrun uses a guided setup wizard to configure clusters, SSH meshes, and hardware detection (such as ConnectX-7 NICs). It manages the distribution of models and containers across cluster nodes via SSH and supports multi-node tensor parallelism. It utilizes a recipe-based system where users can pull optimized configurations from official or community registries hosted on GitHub.

Who it’s for

It is designed for developers and researchers working with NVIDIA DGX Spark hardware who want to quickly deploy LLM inference engines without the overhead of traditional cluster management software.

Highlights

  • Multi-runtime support: Works with vLLM, SGLang, and llama.cpp out of the box.
  • Multi-node tensor parallelism: Automatically detects InfiniBand/RDMA to scale across multiple hosts.
  • VRAM estimation: Predicts if a model will fit in memory before launching.
  • Git-based recipe registries: Access to official, community, and benchmarked recipes for optimized model deployment.
  • Automated setup: Guided wizard for cluster creation, SSH mesh, and system configuration.
  • Model distribution: Automatically syncs models and images to cluster nodes.

Related

  • Project
    eugr/spark-vllm-docker

    A Docker-based deployment framework for vLLM optimized for NVIDIA DGX Spark, enabling high-performance single-node and multi-node cluster inference.

  • Project
    NVIDIA/TensorRT-Edge-LLM

    A high-performance C++ inference runtime for deploying text, vision, audio, and speech models on NVIDIA Jetson, DRIVE, and DGX Spark edge platforms.

  • Project
    NVIDIA/TensorRT-LLM

    An optimization library for LLMs and visual generation models that uses specialized kernels and an efficient runtime to maximize inference throughput and minimize latency on NVIDIA GPUs.

  • Project
    ztxz16/fastllm

    A high-performance C++ inference engine for local LLM deployment that supports hybrid MoE offloading across GPU, CPU, NUMA, and disk.

  • Project
    ai-dynamo/dynamo

    An open-source, datacenter-scale orchestration layer that coordinates multiple inference engines across GPU clusters to maximize throughput and minimize latency for LLMs and multimodal models.