llm-d/llm-d

What it solves

llm-d is designed to eliminate the complexity of tuning and deploying large language models (LLMs) for production at scale. It addresses the performance bottlenecks of standard model servers by providing an orchestration and optimization layer that maximizes throughput and reduces latency across various hardware accelerators.

How it works

llm-d integrates with industry-standard technologies like vLLM and Kubernetes to provide a distributed inference serving stack. It employs several advanced techniques to optimize performance:

• Intelligent Routing: Uses prefix-cache and load-aware balancing, as well as predicted-latency scheduling to decrease latency and increase throughput.
• KV-Cache Management: Implements tiered offloading to CPU or disk and global indexing to increase the effective working set size for multi-turn requests.
• Serving Large Models: Utilizes prefill/decode disaggregation and wide expert-parallelism over fast interconnects to optimize massive models.
• Operational Excellence: Provides intelligent flow control for multi-tenant serving and SLO-aware autoscaling based on real-time signals.
• Batch Processing: Manages large-scale offline inference via OpenAI-compatible Batch APIs and asynchronous processing.

Who it’s for

This project is for engineers and organizations deploying LLMs in production environments on Kubernetes, seeking to achieve state-of-the-art (SOTA) inference performance across different hardware accelerators (such as NVIDIA, AMD, Intel, and Google TPU).

Highlights

• CNCF Sandbox Project: Founded by industry leaders including Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA.
• Significant Performance Gains: Validated gains including 3x higher output throughput and 2x faster TTFT with prefix-cache-aware routing.
• Broad Hardware Support: Optimized for a wide range of accelerators and infrastructure providers.
• Production-Ready Recipes: Provides benchmarked guides and Helm charts to simplify deployment and best practices.