gpustack/gpustack

What it solves

GPUStack simplifies the management of GPU clusters for AI model serving and instance provisioning. It removes the complexity of manually configuring high-performance inference engines and orchestrating resources across diverse environments, such as on-premises servers, Kubernetes, and cloud providers.

How it works

GPUStack acts as an orchestration layer that manages multiple GPU clusters. It automatically configures and deploys pluggable inference engines (like vLLM, SGLang, and TensorRT-LLM) and optimizes them for either low latency or high throughput. The system includes a scheduler that allocates GPUs to maximize resource utilization and provides an OpenAI-compatible API for accessing deployed models.

Who it’s for

It is designed for development teams, IT organizations, and service providers who need to deliver Model-as-a-Service (MaaS) at scale, as well as developers who need on-demand SSH-accessible GPU instances for fine-tuning and interactive workloads.

Highlights

• Multi-Cluster Management: Supports on-premises, Kubernetes, and cloud environments in a single view.
• Pluggable Engines: Rapidly deploy new models using vLLM, SGLang, TensorRT-LLM, or custom engines.
• Performance Optimization: Includes pre-tuned modes and support for speculative decoding (EAGLE3, MTP, N-grams) and extended KV cache systems (LMCache, HiCache).
• Broad Hardware Support: Compatible with NVIDIA, AMD, Ascend, Hygon, MetaX, and other accelerators.
• Enterprise Operations: Features built-in authentication, access control, real-time monitoring via Grafana/Prometheus, and automated failure recovery.