quantumaikr/quant.cpp
LLM inference with 7x longer context. Pure C, zero dependencies. Lossless KV cache compression + single-header library.
What it solves
quant.cpp addresses the memory bottleneck of large context windows on consumer hardware. It eliminates the need for chunking-based Retrieval-Augmented Generation (RAG) for medium-sized documents by allowing users to load entire documents into the model's context window on devices like a 16GB Mac, reducing the risk of "silent hallucinations" that occur when RAG retrieves the wrong document section.
How it works
The project implements high-efficiency KV (Key-Value) cache compression. It uses a "progressive" compression strategy where the most recent ~128 tokens are kept at full FP32 precision while the rest are compressed (up to 6.4x), aligning storage precision with the fact that transformer attention typically concentrates on the most recent tokens. It also includes a specialized 5-stage pipeline called RLV (Read-Locate-Verify) to handle documents that exceed the model's effective working memory cliff.
Who it’s for
Developers and AI researchers who want to run long-context LLMs on consumer-grade laptops (specifically macOS and Linux) without relying on heavy GPU infrastructure or complex RAG pipelines for single-document analysis.
Highlights
- High Compression: Achieves up to 6.4x KV compression, enabling 128K context on a 16GB Mac.
- FP32 Quality: Maintains near-FP32 quality by keeping a small window of tokens at full precision.
- Zero Dependencies: Written in 17.6K lines of C with no external dependencies.
- OpenAI Compatible: Includes a server mode that exposes an OpenAI-compatible HTTP API.
- RLV Pipeline: A Read-Locate-Verify system that bypasses the "working memory cliff" of small models to maintain accuracy on longer texts.