GPUWorks 0.1 Release – Ultra‑lightweight GPU Cluster Manager for vLLM and SGLang

TL;DR

GPUWorks provides a 40 MB binary that creates a single GPU cluster across public clouds, private clouds and Macs, supports NVIDIA, AMD, Huawei Ascend, T‑Head PPU, Hygon and Apple Silicon, and exposes an OpenAI‑compatible API for serving LLMs with vLLM, SGLang and other engines.

One‑click deployment of verified model‑accelerator combos

GPUWorks ships a catalog of 1,235 launch specifications for 392 models (e.g., Qwen, DeepSeek, Llama, Gemma, Whisper). Each spec pins a concrete Hugging Face or ModelScope revision and includes the exact flags needed for a given accelerator. Selecting a model in the UI automatically shows a memory‑estimate, the GPUs chosen, and why other nodes were rejected, then deploys the model with a single click.

"The built‑in catalog ships 1,235 launch specs for 392 models… you pick a model and press deploy instead of hunting for flags." – GPUWorks README

Heterogeneous hardware in a single scheduler

The scheduler treats NVIDIA, AMD Instinct, Huawei Ascend, T‑Head PPU, Hygon DCU and Apple Silicon as first‑class citizens. Nodes report their capabilities via vendor CLIs (nvidia‑smi, npu‑smi, etc.) and the scheduler matches each deployment to a node that satisfies the launch spec. This eliminates the need for separate clusters per vendor.

Unified pool of public and private resources

GPUWorks can ingest instances from RunPod, CompShare, AutoDL, Vast.ai, Alibaba Cloud as well as VMs from ZStack Cloud. After installing the lightweight agent (curl … | sh), the node dials out to the server over a WebSocket, so no inbound firewall rules are required. A single server can thus schedule workloads across rented cloud GPUs and on‑premise servers simultaneously.

Ultra‑lightweight architecture

  • Server and agent are static Go binaries (no CGO, < 40 MB total). The server embeds the web UI and stores all state in a single SQLite file.
  • Outbound WebSocket connection from agents removes the need for port forwarding or VPNs.
  • In a benchmark on Alibaba Cloud, a single server managed 1,003 GPU machines across 7 regions, placed 500+ replicas in 4 minutes, and handled 2,000 concurrent requests while using only 9 CPU cores and 1.6 GB RAM.

Memory‑aware placement and GPU sharing

Before launching, GPUWorks estimates the memory needed for model weights, KV cache and engine overhead. It then enforces hard memory quotas (MiB granularity) so multiple models can share a single GPU (e.g., NVIDIA via HAMi‑core, Ascend via vCANN). The sleep/wake feature lets models take turns on the same card, verified on NVIDIA, Ascend, PPU and Hygon.

Scaling and advanced inference features

  • Replica scaling across nodes, multi‑node tensor/pipeline parallelism, and pre‑fill/ decode disaggregation (e.g., vLLM + NIXL on NVIDIA, vLLM‑Ascend + Mooncake on Ascend).
  • OpenAI‑compatible gateway exposing /v1/chat/completions, /v1/embeddings, etc., with routing strategies (least‑request, prefix‑affinity, failover), API‑key quotas and per‑key token accounting.
  • Observability: real‑time GPU/engine metrics, searchable logs, per‑request OTLP traces, web terminal access, and a built‑in benchmark that sweeps concurrency.
  • AI assistant & MCP: ask the UI questions about the cluster or connect Claude Code (or any MCP client) to a read‑only MCP endpoint for programmatic introspection.

Air‑gapped and offline readiness

GPUWorks supports importing container images and model files from local archives or HTTP file servers, and synchronizing models between nodes without internet access. The usage report can be disabled in settings or via usage_report: false in server.yaml.

Quick start guide (Linux/macOS)

  1. Install the server
    bash <(curl -fsSL https://dl.gpuworks.ai/get.sh)
    The script verifies checksums, installs a systemd service on port 8800, and prints the admin password.
  2. Add a GPU node Run the generated command on the target host, e.g.:
    curl -fsSL http://<server>:8800/install.sh | sh -s -- --server http://<server>:8800 --token <node-token>
  3. Deploy a model In the UI, select Qwen/Qwen3-1.7B, review the memory estimate, and click Deploy. Create an API token and call the model:
    curl http://<server>:8800/v1/chat/completions \
     -H "Authorization: Bearer gw-…" -H "Content-Type: application/json" \
     -d '{"model": "qwen3-1.7b", "messages": [{"role": "user", "content": "Hello!"}]}'

Supported accelerators and features

Accelerator Whole‑GPU deploy dGPU (shared) Sleep/wake Multi‑node PD disaggregation
NVIDIA (CUDA) ✅ Docker, containerd, runveil ✅ MiB‑granular quota ✅ vLLM ✅ ✅ vLLM + NIXL
AMD (ROCm) ✅ Docker, containerd, runveil ✅ User‑space quota ✅ vLLM ✅ vLLM ✅ vLLM + NIXL
Huawei Ascend (CANN) ✅ Docker, containerd, runveil ✅ MiB‑granular quota ✅ vLLM‑Ascend ✅ ✅ vLLM‑Ascend + Mooncake
T‑Head PPU (HGGC) ✅ Docker, containerd, runveil ✅ MiB‑granular quota ✅ vLLM ✅ ✅ SGLang + Mooncake
Hygon DCU (DTK) ✅ Docker, containerd, runveil ✅ MiB‑granular quota ✅ vLLM ✅ 🧪 vLLM + Mooncake
Apple Silicon (M1‑M5) ✅ vllm‑metal, MLX, llama.cpp — — — —

✅ = verified on real hardware, 🧪 = experimental.

Community reaction on Hacker News

  • Positive note – A commenter highlighted the appeal for Mac users: “This tool sounds great for running local LLMs on Mac without the overhead of Kubernetes. Being able to manage clusters with just a lightweight 40 MB binary is impressive.”
  • Clarifying questions – Users asked how public and private clouds are unified and what societal advantages the platform brings, indicating interest but also a need for more explanatory material.

When to consider GPUWorks

  • You need a single OpenAI‑compatible endpoint for models running on diverse hardware.
  • You prefer no Kubernetes and want a binary‑only control plane.
  • Your workload includes both inference and occasional fine‑tuning/training on the same GPUs.
  • You operate in air‑gapped environments or want to import models from private archives.

Limitations and open questions

  • The platform is currently beta‑level; some features (e.g., Hygon Mooncake integration) are marked experimental.
  • Advanced multi‑node orchestration relies on Docker/nerdctl or the built‑in rootless runtime; users without container runtimes must use host processes (macOS).
  • Community documentation is extensive, but newcomers may need step‑by‑step tutorials for mixed‑vendor clusters.

GPUWorks demonstrates that a full‑featured GPU MaaS platform can be delivered in a single 40 MB binary, eliminating the operational overhead of Kubernetes while supporting a wide range of accelerators and cloud providers. Its OpenAI‑compatible gateway and rich observability make it a compelling option for teams looking to serve LLMs at scale across heterogeneous hardware.

Sources