kvmem/kvmem

What it solves

KVMem addresses the memory bottleneck in long-context AI applications by managing conversation history and agent tool data within a constrained GPU working set. It allows users to run large models with massive context windows (e.g., 200K+ tokens) by intelligently offloading and retrieving KV (Key-Value) cache data between GPU VRAM, system RAM, and SSD storage.

How it works

KVMem acts as a management layer for inference engines like llama.cpp and NInfer. It implements a tiered memory architecture where active history stays on the GPU, while idle or less relevant history is moved to host RAM or SSD. It uses LRFU (Least Recently Frequently Used) weighting to decide which parts of the conversation to keep in high-speed memory. When a model needs to reference older parts of a long conversation, KVMem retrieves the required data back into the GPU without requiring a full re-computation of the context.

Who it’s for

This project is for developers and power users running local LLMs on consumer-grade NVIDIA hardware (RTX 20/30/40/50 series) who need to maintain long-running agent sessions or massive document analysis tasks that exceed the VRAM capacity of their graphics cards.

Highlights

• Tiered Memory Management: Automatically moves KV cache between GPU, RAM, and SSD to maximize context capacity.
• Multi-Backend Support: Provides unified management for both llama.cpp and NInfer inference engines.
• Tool & Agent Optimization: Supports complex agent workflows, including tool calling, strict schema constraints, and multi-turn state restoration.
• Hardware-Specific Tuning: Includes specialized Windows packages and launchers for various NVIDIA GPU architectures (sm70 through sm120a).
• Performance Focused: Features adaptive MTP (Multi-Token Prediction), ngram acceleration, and configurable prefill chunking to maintain high token-per-second rates during long-context generation.