gpustack/gpustack

What it solves

GPUStack simplifies the management of GPU resources for AI model serving. It addresses the complexity of orchestrating diverse hardware environments—such as on-premises servers, cloud providers, and Kubernetes clusters—into a unified platform for deploying and scaling AI models.

How it works

GPUStack acts as a central management layer that connects to your GPU-enabled worker nodes. It automatically configures high-performance inference engines like vLLM, SGLang, and TensorRT-LLM to run models efficiently. The platform includes a scheduler that assigns workloads to GPUs to maximize utilization and provides standard APIs for interacting with deployed models. It also monitors system health and performance through integrated dashboards.

Who it’s for

It is designed for development teams, IT organizations, and service providers who need to deliver "Model-as-a-Service" at scale across heterogeneous infrastructure.

Highlights

• Multi-cluster management across on-premises, cloud, and Kubernetes environments.
• Pluggable inference engine architecture supporting vLLM, SGLang, and custom engines.
• Performance-optimized configurations with support for speculative decoding and advanced KV caching.
• On-demand provisioning of SSH-accessible GPU instances for development and fine-tuning.
• Enterprise-grade features including load balancing, automated failure recovery, and detailed token usage metering.
• Broad hardware support including NVIDIA, AMD, Ascend, and several other specialized GPU/NPU accelerators.