mudler/vllm.cpp

a community oriented 1:1, vLLM-alike (Continuous batching, paged KV) engine in C++ with additional features (GGUF, RadixAttention, Cache-aware scheduling, ...)

What it solves

vllm.cpp is a high-performance C++20 inference engine designed to provide the serving capabilities of vLLM without the heavy dependency footprint of Python and PyTorch. It aims to be the smallest possible deployment (a single 66 MiB binary) while maintaining the throughput and feature set of the original vLLM project.

How it works

The engine combines the best features from several leading inference frameworks:

  • vLLM Core: Implements continuous batching, block-paged KV cache, and automatic prefix caching.
  • SGLang: Integrates RadixAttention and cache-aware scheduling.
  • llama.cpp: Adopts a flat C ABI for easy embedding and native support for GGUF quantized files.
  • Hardware Acceleration: Supports a wide range of backends including CUDA, CPU, Metal, and Vulkan from a single source tree.

Who it’s for

It is intended for developers and operators who need to deploy LLMs with minimal overhead, those who want to OpenAI-compatible endpoints without a Python environment, and users running models on diverse hardware (NVIDIA, Apple Silicon, portable GPUs).

Highlights

  • Extreme Lightweightness: 66 MiB binary vs 9.1 GiB vLLM installation.
  • Broad Model Support: 37 registered architectures, including Llama-3, Mistral, Qwen, and DeepSeek.
  • Multimodal Capabilities: Supports image, video, and audio input, as well as video and audio generation via MiniMax-H3.
  • High Performance: Matches or exceeds vLLM throughput on specific models (e.g., Qwen3.6-27B) and outperforms llama.cpp prefill on CPU.
  • Flexible Quantization: Native support for GGUF and NVFP4.
  • C ABI: Provides a versioned C ABI for seamless integration into C, C++, Go, or Rust applications.

Related

  • Project
    ggml-org/llama.cpp

    A high-performance C/C++ implementation for local LLM and VLM inference, supporting a wide range of hardware through quantization and hardware acceleration.

  • Project
    jmaczan/tiny-vllm

    tiny‑vllm is a C++/CUDA LLM inference engine (a smaller, educational version of vLLM) that loads a Llama 3.2 1B Instruct model from safetensors, runs the full forward pass on GPU (prefill + decode), implements KV‑cache, static/continuous batching, FlashAttention‑style online softmax and PagedAttention, and includes a step‑by‑step tutorial for learning or teaching GPU‑based LLM serving.

  • Project
    google/gemma.cpp

    A lightweight, standalone C++ inference engine for Google's Gemma models, designed for research and experimentation with a minimalist, CPU-optimized implementation.

  • Project
    abetlen/llama-cpp-python

    Python bindings for llama.cpp that enable local LLM inference with an OpenAI-compatible API and support for multiple hardware acceleration backends.

  • Project
    kkokosa/dotLLM

    A high-performance LLM inference engine written natively in C#/.NET, providing native CPU/GPU support for transformer models without relying on Python or llama.cpp.