kaito-project/kaito

What it solves

KAITO simplifies the complex process of deploying and managing Large Language Models (LLMs) on Kubernetes. It removes the need for users to manually configure detailed deployment parameters, GPU resource allocation, and storage for model weights, automating the infrastructure orchestration required for AI workloads.

How it works

KAITO uses a Kubernetes Operator pattern with Custom Resource Definitions (CRDs) to manage workloads:

• Workspace: Automates the deployment of an LLM. It estimates GPU memory requirements based on the model and hardware, triggers node auto-provisioning (via Karpenter APIs), and configures the inference engine (currently vLLM) with optimized scheduling parameters.
• InferenceSet: Manages replicas of a model to enable autoscaling based on request load, integrating with KEDA for metric-based scaling.
• InferencePool: Integrates with the Gateway API Inference Extension to enable KVCache-aware routing for efficient request handling.
• RAGEngine: A separate operator that streamlines the deployment of Retrieval Augmented Generation (RAG) services, coordinating the LLM endpoint, embedding services, and vector databases (such as FAISS, Qdrant, or Milvus).

Who it’s for

Platform engineers and DevOps teams who need to deploy LLMs and RAG pipelines on Kubernetes without manually managing the GPU infrastructure and scaling logic.

Highlights

• Automated GPU Provisioning: Uses node auto-provisioners to pick the optimal GPU node count based on accurate memory estimation.
• vLLM Support: Supports any HuggingFace model compatible with vLLM.
• Local NVMe Storage: Leverages built-in local NVMe on GPU nodes for model storage to avoid extra storage overhead.
• Integrated RAG Stack: Provides a complete RAG orchestration using LlamaIndex, including hybrid search (BM25 and vector dense retrieval) via Reciprocal Rank Fusion (RRF).
• Simplified API: Replaces complex deployment parameters with optimized preset configurations for parallelism (PP, DP, TP).