volcano-sh/kthena

What it solves

Kthena is designed to simplify the deployment and management of Large Language Models (LLMs) in production environments. It addresses the complexity of scaling AI infrastructure, optimizing hardware utilization, and managing the model lifecycle within a Kubernetes-native ecosystem, ensuring that inference workloads are scalable, cost-efficient, and reliable.

How it works

Kthena extends Kubernetes using Custom Resource Definitions (CRDs) to manage LLM workloads. It separates the control plane (Kthena-controller-manager) from the data plane (Kthena-router). The controller manager handles the model lifecycle, autoscaling, and scheduling—integrating with the Volcano scheduler for network topology-aware and gang scheduling. The router acts as the entry point for traffic, using pluggable load-balancing algorithms (such as KV-cache aware routing) and supporting advanced patterns like prefill-decode disaggregation, where compute-intensive prefill operations are separated from token generation.

Who it’s for

This platform is intended for platform engineers, AI practitioners, and organizations deploying LLMs at scale in production, who need a cloud-native way to manage multiple inference engines (like vLLM, SGLang, and Triton) and optimize their GPU resources.

Highlights

• Multi-Engine Support: Compatible with vLLM, SGLang, Triton, and TorchServe.
• Prefill-Decode Disaggregation: Optimizes hardware utilization and latency by separating prefill and decode processes.
• Intelligent Routing: Features KV-cache aware and model-load aware routing, along with canary releases and token-based rate limiting.
• Advanced Scheduling: Includes network topology-aware scheduling and gang scheduling for distributed inference groups.
• Cost-Driven Autoscaling: Scales based on CPU, GPU, and memory metrics with configurable budget constraints.
• Dynamic LoRA Management: Allows hot-swapping of adapters without service interruption.