localai-org/vllm.cpp

a community oriented 1:1, vLLM-alike (Continuous batching, paged KV) engine in C++ with additional features (GGUF, RadixAttention, Cache-aware scheduling, ...)

What it solves

vllm.cpp is a high-performance C++20 inference engine designed to be a lightweight, standalone alternative to vLLM. It eliminates the need for Python and PyTorch at inference time, drastically reducing the deployment footprint (from gigabytes to megabytes) while maintaining the same throughput and token-for-token identical output as vLLM.

How it works

It combines the best features of several leading inference engines:

  • vLLM: Adopts the serving core, including continuous batching, block-paged KV cache, and automatic prefix caching.
  • SGLang: Integrates scheduling ideas like RadixAttention and jump-forward decoding.
  • llama.cpp: Implements a flat C ABI and supports GGUF files, allowing computation directly on quantized blocks on the CPU.
  • MLX: Utilizes MLX's GEMM for optimized performance on Apple Silicon.

The engine supports a wide range of hardware (CUDA, CPU, Metal, Vulkan) and quantization formats (NVFP4, EXL3, GGUF) from a single source tree.

Who it’s for

Developers and operators who need to deploy LLMs with minimal overhead, fast cold starts, and high throughput on diverse hardware without the complexity of a Python-based stack.

Highlights

  • Extreme Lightweightness: A 66 MiB binary compared to a 9.1 GiB vLLM installation.
  • vLLM Parity: Token-for-token identical output and matching throughput on supported models.
  • Broad Hardware Support: Native support for CUDA, CPU, Metal, and Vulkan.
  • Broad Model Support: 44 registered architectures including multimodal (image, video, audio) and music generation.
  • Advanced Features: Supports speculative decoding (MTP, DFlash), structured output (JSON schema, GBNF), and external KV offload.
  • GGUF Integration: First-class support for GGUF files with direct computation on compressed blocks on CPU.

Related

  • Project
    vllm-project/tpu-inference

    vLLM JAX (formerly tpu‑inference) is a vLLM plugin that serves JAX and PyTorch LLMs on Google TPUs. It offers a unified backend, high‑performance serving features, and extensive model support, with quick‑start docs and active community support.

  • Project
    leyten/shard

    A protocol and engine for a permissionless compute network that pools VRAM and compute from distributed GPUs to run massive AI models over the open internet.

  • Project
    microsoft/antares

    A compiler solution for PyTorch that enables users to create, benchmark, and optimize custom operators for hardware accelerators like CUDA and ROCm.

  • Project
    dineshsoudagar/local-llms-on-android

    An Android application that enables fully offline, private LLM inference on-device and can act as a local LAN server for other devices via an OpenAI-compatible API.