google/gemma.cpp

lightweight, standalone C++ inference engine for Google's Gemma models.

What it solves

gemma.cpp is a lightweight, standalone C++ inference engine designed for Google's Gemma foundation models. It bridges the gap between complex, deployment-oriented C++ runtimes and high-level Python research frameworks, providing a minimalist implementation that is easy to modify and embed into other projects.

How it works

The engine uses a small core implementation (approximately 2,000 lines of code) to perform CPU-only inference for Gemma 2, Gemma 3, and PaliGemma 2 models. It leverages the Google Highway library for portable SIMD (Single Instruction, Multiple Data) to optimize CPU performance across different architectures.

Key technical features include:

  • Mixed-precision GEMM: Supports fp8, bf16, fp32, and fp64, with automatic runtime autotuning for matrix shapes.
  • Weight Compression: Integrated custom fp8 formats and non-uniform 4-bit (NUQ) compression to reduce memory bandwidth and cache usage.
  • Infrastructure: Implements tensor parallelism with a multi-socket thread pool and supports memory-mapped disk I/O.
  • Frontends: Provides C++ APIs with streaming, a basic interactive CLI, and Python bindings via pybind11.

Who it’s for

It is primarily intended for researchers and developers who need a straightforward, modifiable inference engine for experimentation and research use cases, rather than production-edge deployments.

Highlights

  • Minimalist Design: Small codebase for easy auditing and experimentation.
  • Broad Model Support: Supports text-based Gemma 2-3 and the PaliGemma 2 vision-language model.
  • CPU-Optimized: Uses Highway SIMD for high-performance CPU inference without requiring a GPU.
  • Flexible Integration: Can be incorporated as a library via CMake FetchContent or built as a shared library.
  • Research Tools: Includes a backward pass (VJP) and Adam optimizer for Gemma research.

Related

  • Project
    mudler/vllm.cpp

    A lightweight C++20 inference engine that mirrors vLLM's features and throughput in a single 66 MiB binary, removing the need for Python and PyTorch at inference time.

  • Project
    ggml-org/llama.cpp

    A high-performance C/C++ implementation for local LLM and VLM inference, supporting a wide range of hardware through quantization and hardware acceleration.

  • Project
    microsoft/BitNet

    An official inference framework for 1-bit (ternary) LLMs that enables fast, energy-efficient, and lossless inference on CPUs and GPUs.

  • Dispatch
    Building a High-Performance LLM Inference Engine from Scratch with tiny-vLLM

    An exploration of tiny-vLLM, a C++ and CUDA-based educational project that teaches the internals of LLM inference, from Safetensors loading to PagedAttention.

  • Project
    jmaczan/tiny-vllm

    tiny‑vllm is a C++/CUDA LLM inference engine (a smaller, educational version of vLLM) that loads a Llama 3.2 1B Instruct model from safetensors, runs the full forward pass on GPU (prefill + decode), implements KV‑cache, static/continuous batching, FlashAttention‑style online softmax and PagedAttention, and includes a step‑by‑step tutorial for learning or teaching GPU‑based LLM serving.