gpustack: an open-source GPU cluster manager for AI model serving and instance provisioning

What it solves

GPUStack simplifies the complex process of managing GPU clusters for AI model serving and instance provisioning. It removes the manual effort required to configure high-performance inference engines and orchestrate resources across diverse environments (on-premises, Kubernetes, and cloud providers).

How it works

GPUStack acts as a central manager that orchestrates various pluggable inference engines—such as vLLM, SGLang, and TensorRT-LLM—and schedules GPU resources to maximize utilization. It provides a user interface for deploying models from a catalog, managing worker nodes, and exposing models via OpenAI-compatible APIs. It also supports launching SSH-accessible GPU instances for development and fine-tuning.

Who it’s for

It is designed for development teams, IT organizations, and service providers who need to deliver Model-as-a-Service (MaaS) at scale across multiple GPU clusters.

Highlights

• Multi-Cluster Management: Supports on-premises, Kubernetes, and cloud GPU resources.
• Pluggable Engines: Automatically configures engines like vLLM and SGLang, with support for custom engines.
• Performance Optimization: Includes pre-tuned modes for latency/throughput and support for speculative decoding (EAGLE3, MTP, N-grams) and extended KV cache systems (LMCache, HiCache).
• Broad Accelerator Support: Compatible with NVIDIA, AMD, Ascend NPU, Hygon DCU, and several other specialized AI accelerators.
• Enterprise Operations: Features built-in authentication, access control, real-time monitoring via Grafana/Prometheus, and automated failure recovery.