EricLBuehler/candle-vllm

What it solves

Candle-vLLM provides a high-performance platform for running and serving local Large Language Models (LLMs). It addresses the complexity of deploying production-ready models by offering an OpenAI-compatible API server, a built-in web interface, and advanced memory optimization techniques to handle large models on limited hardware.

How it works

Built on the Candle framework, the project implements several industry-standard inference optimizations:

• Memory Management: Uses PagedAttention and continuous batching to handle multiple requests efficiently. It also features "TurboQuant" for aggressive KV cache compression (down to 2-4 bits), which can extend context windows up to 4.7x.
• Hardware Acceleration: Supports CUDA (Linux) and Metal (macOS) with native Flash Attention, FlashInfer, and CUDA Graphs.
• Scaling: Enables tensor parallelism across multiple GPUs (multi-process or multi-threaded) and across multiple network nodes via TCP-based coordination.
• Quantization: Supports various formats including GGUF, GPTQ, AWQ, Marlin, and FP8/FP4, with the ability to perform "in-situ" quantization on the fly.

Who it’s for

• Developers who need a drop-in OpenAI-compatible backend for their local AI applications.
• ML Engineers looking to deploy large models (like DeepSeek or Qwen) across multi-GPU or multi-node clusters.
• Mac and Linux users who want a unified codebase for high-performance local inference.

Highlights

• OpenAI Compatible: Includes an API server and a ChatGPT-style Web UI.
• Cross-Platform: Native support for both NVIDIA GPUs (CUDA) and Apple Silicon (Metal).
• TurboQuant: Specialized KV cache compression to significantly increase context capacity.
• Distributed Inference: Supports multi-GPU and multi-node setups without requiring MPI.
• Broad Model Support: Compatible with a wide range of models including Llama, Mistral, Qwen, DeepSeek, and GLM.