EricLBuehler/candle-vllm

Efficent platform for inference and serving local LLMs including an OpenAI compatible API server.

What it solves

Candle-vLLM provides a high-performance platform for running and serving local Large Language Models (LLMs). It addresses the complexity of deploying production-ready models by offering an OpenAI-compatible API server, a built-in web interface, and advanced memory optimization techniques to handle large models on limited hardware.

How it works

Built on the Candle framework, the project implements several industry-standard inference optimizations:

  • Memory Management: Uses PagedAttention and continuous batching to handle multiple requests efficiently. It also features "TurboQuant" for aggressive KV cache compression (down to 2-4 bits), which can extend context windows up to 4.7x.
  • Hardware Acceleration: Supports CUDA (Linux) and Metal (macOS) with native Flash Attention, FlashInfer, and CUDA Graphs.
  • Scaling: Enables tensor parallelism across multiple GPUs (multi-process or multi-threaded) and across multiple network nodes via TCP-based coordination.
  • Quantization: Supports various formats including GGUF, GPTQ, AWQ, Marlin, and FP8/FP4, with the ability to perform "in-situ" quantization on the fly.

Who it’s for

  • Developers who need a drop-in OpenAI-compatible backend for their local AI applications.
  • ML Engineers looking to deploy large models (like DeepSeek or Qwen) across multi-GPU or multi-node clusters.
  • Mac and Linux users who want a unified codebase for high-performance local inference.

Highlights

  • OpenAI Compatible: Includes an API server and a ChatGPT-style Web UI.
  • Cross-Platform: Native support for both NVIDIA GPUs (CUDA) and Apple Silicon (Metal).
  • TurboQuant: Specialized KV cache compression to significantly increase context capacity.
  • Distributed Inference: Supports multi-GPU and multi-node setups without requiring MPI.
  • Broad Model Support: Compatible with a wide range of models including Llama, Mistral, Qwen, DeepSeek, and GLM.

Related

  • Project
    ggml-org/llama.cpp

    A high-performance C/C++ implementation for local LLM and VLM inference, supporting a wide range of hardware through quantization and hardware acceleration.

  • Project
    mlc-ai/web-llm

    A high-performance in-browser LLM inference engine that uses WebGPU to run language models locally without a server.

  • Project
    InternLM/lmdeploy

    A toolkit for compressing, deploying, and serving LLMs and VLMs, offering high-performance inference engines and quantization to maximize throughput.

  • Project
    xLLM-AI/xllm

    xLLM is an efficient LLM inference framework optimized for Chinese AI accelerators, designed for high-throughput, low-latency enterprise-grade deployment.

  • Project
    Zefan-Cai/KVCache-Factory

    A unified playground for implementing and evaluating KV cache compression, retrieval, merging, and quantization methods to optimize long-context LLM inference memory usage.