spiritbuun/buun-llama-cpp

Experimental llama.cpp fork for inference research and development

What it solves

buun-llama-cpp is an experimental research fork of llama.cpp designed to maximize LLM inference efficiency on VRAM-constrained hardware. It specifically addresses the trade-off between context length, memory usage, and model quality by introducing dynamic KV cache quantization and advanced MoE (Mixture-of-Experts) caching strategies.

How it works

The project implements several key technical innovations:

  • Variable Bit-Rate (VBR) KV Cache: Instead of a fixed quantization level, VBR dynamically quantizes the KV cache as the session grows. It starts at FP16 and degrades layer-by-layer (using a predefined "ladder" of codecs) only when VRAM pressure requires it, ensuring the highest possible quality for the current context depth.
  • Trellis-Coded Quantization (TCQ): A specialized codec that uses a 512-state trellis and Viterbi encoding to significantly reduce KL-divergence (error) at very low bit rates (2-3 bits) compared to standard scalar quantization.
  • MoE Caching: For massive MoE models (like DeepSeek V4 Flash), it allows routed experts to reside in system RAM while caching "hot" expert tensors in spare VRAM to accelerate inference.
  • GPU-Swap for Vision: To save VRAM, it can temporarily unload speculative decoding components to load a vision encoder (mmproj) onto the GPU only when an image is processed, then swap back.

Who it’s for

Developers and researchers working with large models on consumer GPUs (e.g., RTX 3090) who need to push the limits of context window size and inference speed without sacrificing too much model accuracy.

Highlights

  • Dynamic Quality Control: VBR allows the cache to degrade gracefully from FP16 down to 1.25 bits/value.
  • High-Efficiency Codecs: TCQ provides near-FP16 perplexity at 3.25 bits/value.
  • Adaptive MoE Management: Automatic fitting and expert-parallel dispatch for large-scale MoE models.
  • VRAM Optimization: Intelligent swapping between vision encoders and speculative decoding sidecars.

Related

  • Dispatch
    vLLM DeepSeek V4 Support: Efficient Long-context Attention

    vLLM now supports DeepSeek V4-Pro and V4-Flash, implementing a new attention mechanism that enables context lengths up to one million tokens with significant KV cache memory savings.

  • Project
    ikawrakow/ik_llama.cpp

    A high-performance fork of llama.cpp that provides better CPU inference speed and advanced quantization types for running LLMs on consumer hardware.

  • Project
    NVIDIA/kvpress

    A framework for implementing and benchmarking KV cache compression methods to reduce memory usage and increase inference speed for long-context LLMs.

  • Dispatch
    vLLM TurboQuant Study: Accuracy and Performance Analysis

    A comprehensive study by vLLM reveals that FP8 remains the superior default for KV-cache quantization, while TurboQuant variants trade significant throughput and latency for additional memory capacity.

  • Dispatch
    vLLM FP8 KV-Cache and Attention Quantization Update

    vLLM has optimized FP8 KV-cache and attention quantization to reduce decode latency and memory usage while maintaining near-baseline accuracy across Hopper and Blackwell architectures.