bytedance/InfiniStore

InfiniStore – High‑Performance KV Store for LLM Inference

What it is

• An open‑source key‑value (KV) store built to accelerate large‑language‑model (LLM) inference clusters.
• Designed for two common deployment patterns:
  1. Prefill‑decoding disaggregated clusters – separate nodes handle the heavy pre‑fill work and the fast decoding work; InfiniStore moves and re‑uses KV caches between them.
  2. Non‑disaggregated clusters – every node does both pre‑fill and decoding; InfiniStore acts as a shared, extra‑large KV cache pool that sits alongside GPU and CPU caches.
• Works today with the vLLM inference engine via the LMCache integration; support for SGLang and other engines is in progress.

Why it matters

• LLM inference relies on a KV cache that stores intermediate attention keys/values. Keeping this cache on‑device is fast but limited by GPU memory. InfiniStore lets you spill, transfer, and reuse that cache across nodes, reducing GPU memory pressure and latency when scaling to many requests.
• Provides low‑latency network paths (TCP, RoCE, InfiniBand) and can run on either GPU‑enabled or CPU‑only machines.

Key capabilities

Feature	What it does
KV cache transfer	Moves cached attention data from a prefill node to a decoding node so the decoding step can start instantly.
KV cache reuse	Allows a later request to reuse a previously stored cache (e.g., for repeated prompts), cutting compute time.
Cross‑node cache pool	Acts as a central, large‑capacity cache that all nodes can read/write, extending the effective cache beyond a single GPU’s memory.
Network flexibility	Supports plain TCP/IP as well as high‑performance RDMA (RoCE or InfiniBand) for sub‑microsecond latency.
Standalone mode	Can be run as a generic KV store for any LLM training or inference service, not just vLLM.

Typical workflow

1. Start a server on a machine (GPU or CPU) with the appropriate network flags, e.g.

   infinistore --service-port 12345               # TCP
   infinistore --service-port 12345 --dev-name mlx5_0 --link-type Ethernet  # RoCE
   infinistore --service-port 12345 --dev-name mlx5_0 --link-type IB       # InfiniBand
   

2. Run a client (synchronous or async examples are provided in infinistore/example/).
3. In a vLLM cluster, install vLLM, LMCache, and InfiniStore on every node; the inference engine will automatically call InfiniStore to fetch or store KV caches.

Installation

• Quick install (most users):

  pip install infinistore
  

• From source (for contributors or custom builds):

  apt install libuv1-dev libflatbuffers-dev libspdlog-dev libfmt-dev \
      ibverbs-utils libibverbs-dev libboost-dev libboost-stacktrace-dev
  pip install --no-build-isolation -e .
  pip install pre-commit && pre-commit install
  

• Verify with:

  infinistore --manage-port 8088
  curl http://127.0.0.1:8088/selftest
  

Getting involved

• Run the test suite: pytest infinistore/test_infinistore.py.
• Follow the pre‑commit style checks before submitting PRs.
• Contributions can be code, documentation, or other improvements.

Where to learn more

• Docs site: https://bytedance.github.io/InfiniStore/
• Slack community (invite link in README).
• Source code and issue tracker on GitHub.

---

InfiniStore is a niche but crucial piece of the LLM inference stack, handling the high‑throughput, low‑latency cache sharing that lets large models serve many requests efficiently.