vllm-project/production-stack

What it solves

This project provides a reference implementation for deploying vLLM in production environments. It solves the challenge of scaling from a single instance to a distributed deployment without requiring changes to application code, while providing built-in monitoring and performance optimizations like request routing and KV cache offloading.

How it works

The stack is deployed via Helm on Kubernetes and consists of three primary components:

• Serving Engine: Multiple vLLM engines that run the LLMs.
• Request Router: A layer that directs traffic to the most appropriate backend based on routing keys or session IDs to maximize KV cache reuse.
• Observability Stack: A monitoring system using Prometheus and Grafana to track metrics such as latency, Time-to-First-Token (TTFT), and GPU KV cache usage.

Who it’s for

Developers and ML engineers who need to move vLLM-based LLM serving from a single-node setup to a scalable, production-ready Kubernetes cluster across various cloud platforms (AWS, GCP, Azure, Lambda Labs).

Highlights

• Seamless Scaling: Scale vLLM instances without modifying application code.
• Intelligent Routing: Supports round-robin and session-ID based routing to improve efficiency.
• OpenAI API Compatibility: Maintains the same interface as vLLM for easy integration.
• Comprehensive Monitoring: Detailed Grafana dashboards for tracking request latency, pending requests, and KV cache hit rates.
• KV Cache Offloading: Integration with LMCache to optimize performance.