vllm-project/aibrix
Cost-efficient and pluggable Infrastructure components for GenAI inference
AIBrix – Quick Overview
AIBrix is an open‑source, cloud‑native platform for running large language model (LLM) inference at scale. It is built to run on Kubernetes and provides a collection of components that handle the whole lifecycle of a GenAI service: model download, routing, autoscaling, fault detection, and cost‑aware heterogeneous GPU scheduling.
What it does
- Deploys LLMs (including LoRA‑adapted variants) as Kubernetes resources.
- Routes requests to the appropriate model replica via an LLM‑aware gateway.
- Autosscales inference pods based on real‑time traffic and service‑level objectives.
- Shares KV‑cache across nodes to reduce recomputation for long‑context prompts.
- Detects GPU failures and can gracefully retire faulty hardware.
- Supports mixed‑GPU clusters, letting you run cheaper GPUs for low‑load traffic while keeping high‑end GPUs for latency‑critical requests.
Core pieces
| Component | Role |
|---|---|
| Unified AI Runtime (sidecar) | Standardises metrics, handles model download, and provides a thin management API. |
| LLM Gateway & Routing | Entry point for client calls; balances traffic across model replicas. |
| App‑Tailored Autoscaler | Extends Kubernetes HPA with LLM‑specific signals (e.g., token‑throughput, queue depth). |
| Distributed KV Cache | Shares attention cache across pods, enabling high‑density serving of long contexts. |
| CRDs & Operator | Declarative Kubernetes objects (e.g., LLMModel, LLMDeployment) that the operator reconciles. |
| GPU Failure Detector | Monitors hardware health and triggers pod rescheduling on faults. |
Who might use it
- Enterprises that need a production‑grade, Kubernetes‑native way to serve many LLMs while keeping costs under control.
- Research labs that want a reproducible, scalable inference stack for system‑level experiments.
- Cloud providers or platform teams looking to expose a managed GenAI service to internal users.
Getting started (very short)
# Clone repo
git clone https://github.com/vllm-project/aibrix.git && cd aibrix
# Install CRDs and dependencies (nightly example)
kubectl apply -k config/dependency --server-side
kubectl apply -k config/crd --server-side
kubectl apply -k config/default
Or use the released YAMLs for a stable version (e.g., v0.8.0‑rc.1).
Documentation & community
- Full docs: https://aibrix.readthedocs.io/latest/
- Blog, white‑paper, and release notes linked from the README.
- Community Slack:
#aibrix(via the vLLM dev workspace).
License
Apache 2.0 – free for commercial and academic use.
Bottom line: AIBrix packages the plumbing needed to run LLM inference on Kubernetes, handling model management, routing, autoscaling, cache sharing, and hardware reliability, all under an Apache‑2.0 licence.
Related
- Project
- Project
- Project
- Project
- Project