volcano-sh/kthena

Lightweight, Modular, Kubernetes-native AI serving platform for scalable model serving.

What it solves

Kthena is a Kubernetes-native platform designed to simplify the deployment, scaling, and management of Large Language Model (LLM) inference workloads in production. It addresses the complexity of managing diverse inference engines and optimizing hardware utilization through advanced routing and scheduling policies.

How it works

Kthena splits its architecture into two decoupled, independently deployable planes:

  • Control Plane (Kthena-controller-manager): Manages the LLM inference lifecycle using Kubernetes Custom Resource Definitions (CRDs). It handles deployment, scaling, and upgrades, integrating with the Volcano scheduler for advanced scheduling policies like gang scheduling.
  • Data Plane (Kthena-router): Acts as the entry point for inference traffic, routing requests based on model name, headers, or URI patterns. It supports advanced load-balancing algorithms and specific routing for prefill-decode disaggregation.

Who it’s for

Platform engineers and AI practitioners who need an enterprise-grade, scalable LLM serving platform that integrates with existing Kubernetes-native cloud-native patterns and supports multiple inference engines.

Highlights

  • Modular Design: The platform consists of two self-contained Go binaries that can be deployed independently (workload controllers or router).
  • Prefill-Decode Disaggregation: Optimizes hardware utilization and latency by separating compute-intensive prefill operations from token generation (decode) processes.
  • Multi-Engine Support: Compatible with vLLM, SGLang, Triton, and TorchServe.
  • Intelligent Routing: Features KV-cache aware and model load aware load-balancing, as well as support for canary releases and token-based rate limiting.
  • Advanced Scheduling: Includes network topology-aware scheduling and gang scheduling to prevent resource waste in distributed inference groups.
  • Cost-Driven Autoscaling: Scales based on CPU, GPU, memory, and custom metrics with configurable budget constraints.

Related

  • Project
    vllm-project/aibrix

    AIBrix is an open‑source, Kubernetes‑native platform that provides a full stack for deploying, routing, autoscaling, and managing large language model inference, including LoRA support, distributed KV‑cache, heterogeneous GPU scheduling, and GPU‑failure detection, all under Apache 2.0.

  • Project
    vllm-project/production-stack

    A reference implementation for production vLLM deployment on Kubernetes, providing a scalable inference stack with request routing and observability.

  • Project
    kaito-project/kaito

    A Kubernetes operator suite that automates the deployment, scaling, and management of LLM inference, fine-tuning, and RAG engines.

  • Project
    InftyAI/llmaz

    A production-ready LLM inference platform for Kubernetes that simplifies deployment, scaling, and management across various inference backends.

  • Project
    ome-projects/ome

    OME (Open Model Engine) is a Kubernetes operator that lets you declaratively manage and serve large language models. It provides custom resources for models, runtimes, GPU classes, and inference services, automatically selects the best runtime, schedules GPU resources, supports advanced rollout and autoscaling patterns, and includes a CLI and web console for operations.