ikawrakow/ik_llama.cpp

llama.cpp fork with additional SOTA quants and improved performance

What it solves

ik_llama.cpp is a high-performance fork of llama.cpp designed to improve CPU inference performance and introduce state-of-the-art quantization methods. It addresses the limitations of the mainline version by providing faster prompt processing on CPUs and expanding the range of supported quantization types for better model efficiency.

How it works

The project implements a variety of advanced quantization kernels and optimizations. It uses specialized CPU backends (AVX2, AVX-512, and ARM_NEON) and CUDA (Turing or newer) to accelerate matrix multiplications. Key technical implementations include Trellis quants, IQK quants, and Hadamard transforms for K-cache and V-cache, alongside a "graph" split mode for multi-GPU setups and auto-fit VRAM offloading for MoE and dense models.

Who it’s for

Users who want to run Large Language Models (LLMs) with higher efficiency on consumer hardware, particularly those relying on CPU-heavy inference or hybrid CPU/GPU setups, and developers who need early access to new inference features like MTP decoding and fused delta-nets.

Highlights

  • Enhanced CPU Performance: Significant speedups in prompt processing for non-interleaved quants and MoE models.
  • Advanced Quantization: Support for Trellis quants (IQ1_KT to IQ4_KT), IQK quants, and MXFP4.
  • Broad Model Support: Compatibility with a wide array of modern models including DeepSeek-V3/V4, Qwen3, Gemma 4, and Llama-4.
  • Inference Features: Includes MTP decoding, self-speculative decoding, and a built-in WebUI with MCP support.
  • Hardware Optimization: Specialized support for AVX-512 (Zen4/Sapphire Rapids+) and CUDA Turing+ GPUs.

Related

  • Project
    ggml-org/llama.cpp

    A plain C/C++ implementation for high-performance LLM and VLM inference across a wide range of hardware with minimal setup.

  • Project
    spiritbuun/buun-llama-cpp

    An experimental llama.cpp fork that introduces Variable Bit-Rate KV cache quantization and Trellis-Coded Quantization to maximize context length and quality on VRAM-constrained GPUs.

  • Project
    charlie12345/ROCmFPX

    An extension for llama.cpp that introduces AMD-optimized GGUF quantization formats and accelerated kernels to improve LLM inference speed and memory efficiency on Radeon GPUs.

  • Project
    alibaba/rtp-llm

    A high-performance LLM inference acceleration engine developed by Alibaba that optimizes throughput and latency for production-scale deployments.

  • Dispatch
    llama.cpp and llama.app: Local LLM Inference and Ecosystem

    llama.cpp and its official home llama.app provide high-performance, hardware-agnostic local LLM inference, now featuring a simplified 'llama serve' command and integration with local coding agents like Pi.