EricLBuehler/candle-vllm

Efficent platform for inference and serving local LLMs including an OpenAI compatible API server.

What it solves

Candle-vLLM provides a high-performance platform for running and serving local Large Language Models (LLMs). It addresses the complexity of deploying production-ready inference servers by offering an OpenAI-compatible API, a built-in web interface, and advanced memory optimization techniques to handle large models on available hardware.

How it works

Built on the Candle framework, the project implements several industry-standard optimization techniques to maximize throughput and reduce memory usage:

  • Memory Management: Uses PagedAttention for efficient KV cache management and supports "TurboQuant" for aggressive 2-4 bit KV cache compression, extending context windows up to 4.7x.
  • Execution Optimizations: Employs continuous batching, CUDA Graphs, and native Flash Attention/FlashInfer to speed up token generation.
  • Hardware Flexibility: Supports CUDA (Linux) and Metal (macOS), allowing the same codebase to run across different platforms.
  • Scalability: Enables tensor parallelism across multiple GPUs (multi-process or multi-threaded) and distributes inference across multiple machines via TCP-based coordination.
  • Quantization: Supports various formats including GGUF, GPTQ, AWQ, Marlin, and FP8/FP4, with the ability to perform in-situ quantization on the fly.

Who it’s for

This tool is designed for developers and researchers who need to serve LLMs locally with production-grade performance, specifically those targeting NVIDIA GPUs or Apple Silicon.

Highlights

  • OpenAI Compatible: Includes an API server and a ChatGPT-style Web UI for immediate usability.
  • Cross-Platform: Native support for both CUDA and Metal.
  • Advanced Compression: TurboQuant KV cache compression significantly reduces memory overhead.
  • Speculative Decoding: Supports both built-in Multi-Token Prediction (MTP) and external DFlash2 draft models to accelerate generation.
  • Enterprise Scaling: Multi-node inference capabilities without the need for MPI.

Related

  • Project
    mlc-ai/web-llm

    A high-performance in-browser LLM inference engine that uses WebGPU to run language models locally without server-side processing.

  • Project
    ztxz16/fastllm

    FastLLM is a C++‑based, PyTorch‑free inference engine for large language models. It supports dense and MoE models, hybrid GPU/CPU/NUMA/disk deployment, multi‑GPU tensor parallelism, speculative decoding (MTP/DSpark/DFlash2), many quantization formats, and offers a Python CLI, web UI, and OpenAI‑compatible API server. Install via `pip install ftllm`, run a model with `ftllm run`, or launch a service with `ftllm server`. Extensive docs cover model‑specific guides, performance tuning, and community support.

  • Project
    runpod-workers/worker-vllm

    A worker for Runpod Serverless that enables easy deployment of vLLM-powered LLMs with a fully OpenAI-compatible API interface.

  • Project
    Zefan-Cai/KVCache-Factory

    A unified playground for implementing and evaluating KV cache compression, retrieval, merging, and quantization methods to optimize long-context LLM inference memory usage.

  • Project
    InternLM/lmdeploy

    A toolkit for compressing, deploying, and serving LLMs and VLMs, offering high-performance inference engines and quantization to maximize throughput.