spiritbuun/buun-llama-cpp

Experimental llama.cpp fork for inference research and development

buun‑llama‑cpp – Experimental Llama.cpp Fork

What it is – A research‑oriented fork of the popular llama.cpp inference engine. It adds a set of new KV‑cache quantisation techniques that try to keep the quality of the model’s attention memory while using far less GPU memory. The project is aimed at people who already run LLM inference on a single GPU (or a multi‑GPU server) and want to push the context length or fit larger models without sacrificing too much accuracy.

Core innovations

Feature What it does Why it matters
VBR – Variable‑Bit‑Rate KV cache Dynamically compresses the key‑value cache layer‑by‑layer as the prompt grows, starting at FP16 and degrading to a ladder of custom codecs (turbo8 → turbo4 → turbo3_tcq → … → turbo1_tcq). Gives you the longest possible context for the VRAM you have, while keeping the highest‑quality representation for the early part of the prompt.
Turbo‑quant codecs A family of fast, low‑bit quantisers (turbo8, turbo4, turbo3, turbo2, turbo1) plus trellis‑coded variants (*_tcq). Reduce KV memory to 1‑4 bits per value with only a tiny loss in perplexity; the trellis versions cut KL‑divergence ~40 % compared with scalar quantisation.
TCQ – Trellis‑Coded Quantisation Uses a 512‑state trellis and Viterbi encoding to get a larger effective codebook at the same bit‑rate. Improves quality at very low bit‑rates (e.g., 3.25 bpv) while keeping decode speed close to FP16 on modern GPUs.
Native .safetensors & EXL3 support Loads models stored as .safetensors (including EXL3 quantised weights) directly, no conversion to GGUF needed. Simplifies the workflow for users of newer quantisation formats.
Speculative decoding + vision swap (--mmproj-gpu-swap) Moves the vision encoder (mmproj) to CPU until an image is actually needed, then loads it to GPU for a quick encode. Allows multimodal models to run on GPUs that would otherwise be out‑of‑memory.
MoE cache handling Automatic placement of mixture‑of‑experts (MoE) expert weights between GPU and system RAM, with adaptive “auto” mode. Makes very large MoE models (e.g., DeepSeek‑V4‑Flash) usable on consumer‑grade hardware.

How you use it

The fork works like the upstream llama.cpp binary. A typical launch on a dedicated GPU is:

llama-server -m model.gguf          # VBR is on by default
  • To force the full codec ladder (allowing the cache to degrade to the lowest tier) add -ct vbr.
  • To limit the context length and let VBR spend all VRAM on that window: -c 30000.
  • To manually set a VRAM budget: --vbr-vram 8G.
  • To start with a lower‑quality entry tier (useful on very tight memory): --vbr-entry t8.
  • For multimodal work, enable the swap: --mmproj-gpu-swap.
  • For MoE models, the recommended flags are --fit on --moe-cache auto.

All of the knobs are optional – the defaults aim for the best quality‑to‑memory trade‑off automatically.

Performance highlights (from the README)

Codec Bits/value Perplexity (PPL) Prefill speed (tokens/s) Decode speed (tokens/s)
f16 (reference) 16.0 5.683 1104 30.7
turbo3_tcq (trellis) 3.25 5.668 818 26.6
turbo3 (scalar) 3.5 5.730 1015 28.1
turbo1_tcq (1.25 bpv) 1.25 6.012 991 28.5

The trellis‑coded codecs achieve near‑FP16 quality while using a fraction of the memory, and on a RTX 3090 the decode speed stays within a few tokens per second of the FP16 baseline.

Who should care?

  • Researchers and hobbyists who want to experiment with longer contexts (20k‑30k tokens) on a single GPU.
  • Deployers of multimodal or MoE models that otherwise exceed the VRAM of a consumer card.
  • Anyone interested in KV‑cache quantisation – the repo includes the codebooks and training scripts for the trellis codecs.

Limitations / caveats

  • The fork is labelled highly experimental – expect occasional crashes or incompatibilities with newer llama.cpp releases.
  • Requires a CUDA or ROCm build with flash‑attention enabled; CPU‑only inference is not supported for the Turbo codecs.
  • Some advanced features (e.g., tier‑aware save‑restore, speculative‑decode checkpointing) are still work‑in‑progress.

Quick start checklist

  1. Build with CUDA/ROCm and flash‑attention (make LLAMA_CUDA=1).
  2. Place a GGUF or supported .safetensors model in the working directory.
  3. Run llama-server -m model.gguf.
  4. Add -v to watch VBR degradation steps, or -ct vbr / --vbr-floor t4 to fine‑tune.
  5. For vision, add --mmproj-gpu-swap and supply the mmproj.gguf file.

Bottom line – buun‑llama‑cpp is a serious, research‑focused extension of llama.cpp that pushes the limits of KV‑cache memory efficiency through variable‑bit‑rate quantisation and trellis‑coded codecs. If you are already comfortable with llama.cpp and need more context or want to run large MoE/vision models on limited GPU memory, this fork gives you a concrete, well‑documented set of tools to do so.

Related

  • Project
    ikawrakow/ik_llama.cpp

    A high-performance fork of llama.cpp that introduces advanced quantization types, experimental model support, and optimized inference kernels for CPU and GPU.

  • Project
    charlie12345/ROCmFPX

    An extension for llama.cpp that introduces AMD-optimized GGUF quantization formats and accelerated kernels to improve LLM inference speed and memory efficiency on Radeon GPUs.

  • Project
    localai-org/apex-quant

    A MoE-aware mixed-precision quantization technique for llama.cpp that reduces MoE model size and increases speed while maintaining high accuracy.

  • Project
    sudoingX/qwen38-mtp

    A community guide and benchmark collection for enabling MTP speculative decoding in llama.cpp to speed up Qwen3.8-27B inference on consumer GPUs, with a single flag and tuning rules.

  • Project
    quantumaikr/quant.cpp

    A high-performance C library for LLM inference that uses KV cache compression to enable long-context document understanding on consumer hardware, reducing the need for chunk-based RAG.