cchenax/px-kvstore

Key-Value Store Solution for AI

What it solves

PX-KVStore is a high-performance, in-memory key-value store designed to eliminate the overhead of heavy production databases during AI experimentation and local inference. It specifically addresses the need for deterministic LLM caching—ensuring that identical prompts and parameters result in consistent cache hits—and provides a low-latency storage layer for AI engineering workflows.

How it works

The engine uses a sharded architecture with consistent hashing and virtual nodes to distribute data evenly across CPU cores and avoid "hot shards." It employs a layered approach to data management:

  • Storage & Retrieval: Uses a hash map for O(1) lookups and pluggable LRU/LFU eviction policies. It supports the Redis protocol for compatibility with existing clients.
  • Durability: Combines a Write-Ahead Log (WAL) for zero-data-loss recovery and atomic snapshots for fast startup.
  • AI-Specific Logic: Implements a deterministic cache-key lifecycle that normalizes prompts, canonicalizes LLM parameters, and hashes the combination (prompt, model, params, version) using SHA-256.
  • Replication: Uses an asynchronous Leader-Follower model with initial full syncs and incremental updates.
  • Observability: Integrates Prometheus metrics and a JSON API to track latency and system health.

Who it’s for

AI engineers and researchers who need a lightweight, fast, and reproducible caching layer for LLM inference and local AI experimentation without the complexity of a full-scale distributed database.

Highlights

  • Deterministic LLM Caching: Built-in normalization and canonicalization for prompt-based cache keys.
  • Redis Protocol Support: Compatible with standard Redis clients and supports Redis Streams.
  • Consistent Hashing: Scalable sharding that minimizes data re-mapping during scaling.
  • High Performance: Capable of reaching 138k QPS with low p95 latency.
  • Advanced Monitoring: Includes Count-Min Sketch for top-K heavy hitter detection and per-namespace hot-key reporting.

Related

  • Project
    LMCache/LMCache

    A KV cache management layer for LLM inference that reduces TTFT and improves throughput by making KV caches persistent and reusable across requests and engines.

  • Project
    ovg-project/kvcached

    A KV cache library that brings virtual memory abstraction to LLM serving, enabling elastic and demand-driven GPU memory sharing across multiple models.

  • Project
    jundot/omlx

    An optimized LLM inference server for Apple Silicon Macs featuring tiered KV caching, continuous batching, and a native macOS menu bar management app.

  • Project
    ModelEngine-Group/unified-cache-management

    A cache management framework for LLMs that reduces inference latency and GPU memory usage by persisting KV caches and implementing sparse attention retrieval.

  • Project
    Zefan-Cai/KVCache-Factory

    A unified playground for implementing and evaluating KV cache compression, retrieval, merging, and quantization methods to optimize long-context LLM inference memory usage.