gpustack/gpustack

What it solves

GPUStack simplifies the management of GPU clusters for AI model serving and instance provisioning. It removes the complexity of manually configuring inference engines and managing hardware across different environments (on-premises, Kubernetes, and cloud), allowing teams to deploy and scale AI models as a service.

How it works

GPUStack acts as an orchestration layer that manages multiple GPU clusters. It automatically selects and configures high-performance inference engines like vLLM, SGLang, and TensorRT-LLM, or allows for custom engines. The system includes a scheduler that allocates GPUs to maximize resource utilization and provides an OpenAI-compatible API for accessing deployed models.

Who it’s for

It is designed for development teams, IT organizations, and service providers who need to deliver Model-as-a-Service at scale across diverse hardware accelerators (NVIDIA, AMD, Ascend, etc.).

Highlights

• Multi-Cluster Management: Supports on-premises, Kubernetes, and cloud environments.
• Pluggable Engines: Automated configuration of vLLM, SGLang, and TensorRT-LLM with support for custom engines.
• Performance Optimization: Built-in support for speculative decoding (EAGLE3, MTP, N-grams) and extended KV cache systems (LMCache, HiCache).
• On-Demand Instances: Launches SSH-accessible GPU instances for development and fine-tuning.
• Broad Hardware Support: Compatible with a wide range of accelerators including NVIDIA, AMD, Ascend NPU, and others.
• Enterprise Operations: Includes automated failure recovery, load balancing, monitoring (Grafana/Prometheus), and access control.