kubeai-project/kubeai

AI Inference Operator for Kubernetes. The easiest way to serve ML models in production. Supports VLMs, LLMs, embeddings, and speech-to-text.

What it solves

KubeAI simplifies the deployment and scaling of machine learning models on Kubernetes. It addresses the performance bottlenecks of standard Kubernetes load balancing (which is often inefficient for stateful LLM KV caches) and removes the need for complex external dependencies like Istio or Knative to achieve features like scale-from-zero.

How it works

KubeAI uses two main components:

  1. Model Proxy: An OpenAI-compatible API gateway that implements a prefix-aware load balancing strategy to optimize KV cache utilization and handles request queueing and retries.
  2. Model Operator: A Kubernetes operator that manages backend server pods, automating model downloads, volume mounting, and the orchestration of LoRA adapters.

Who it’s for

It is designed for developers and platform engineers who need to run LLMs, embedding models, and speech-to-text models on Kubernetes across various hardware (CPU, GPU, TPU) without managing complex infrastructure overhead.

Highlights

  • Prefix-aware Load Balancing: Optimizes performance for vLLM by improving KV cache hits.
  • Scale-from-Zero: Automatically spins up model pods based on demand without requiring Knative.
  • OpenAI Compatible: Works seamlessly with existing OpenAI client libraries.
  • Hardware Flexible: Supports CPU, GPU, and TPU deployments.
  • Broad Model Support: Operates vLLM, Ollama, FasterWhisper, and Infinity servers.