openlake-project/openlake

OpenLake is a high performance storage engine for efficient LLM inference and GPU Training

OpenLake – Fast, Distributed Storage for LLM Inference & Training

What it is

  • OpenLake is an open‑source storage engine written in Rust, built on Linux io_uring and RDMA. It is designed to keep data movement between NVMe/SSD, network, and GPU memory ultra‑low‑latency, targeting the massive KV‑cache and checkpointing needs of large language model (LLM) workloads.

Why it matters

  • LLM inference and training spend a lot of time waiting for small, random reads/writes (e.g., KV‑cache look‑ups, checkpoint shards). OpenLake promises million‑plus IOPS and sub‑millisecond latency, which translates into up to 8× higher throughput and 66× faster time‑to‑first‑token in the authors’ benchmarks.

Key capabilities

Feature What it does
KV‑Cache Offload Stores petabyte‑scale key‑value cache on a distributed pool of GPU‑host nodes, letting many GPUs share the same cache and avoid repeated pre‑fill work.
VectorDB support Fast index building and vector serving directly from the storage layer.
Ultra‑fast checkpointing Low‑latency object store for RL and training checkpoints, reducing GPU idle time.
Model‑training I/O Optimised for small‑file random reads/writes, cutting training‑time overhead.
Context storage Persists massive conversation histories for retrieval‑augmented agents.
Zero‑copy GPUDirect With RDMA or GPUDirect‑Storage, data can move straight from NVMe/ NIC to GPU memory without staging in host RAM.
Burst‑aware RDMA (Pac​edRDMA) Credit‑based flow control prevents network congestion during request spikes, keeping tail latency low.
Erasure‑coded durability SIMD Reed‑Solomon coding spreads data + parity across drives, offering fault tolerance with less overhead than full replication.

Typical use‑cases

  • Scaling KV‑cache for multi‑GPU inference servers (e.g., vLLM, DeepSpeed).
  • Providing a high‑throughput object store for checkpoint files in large‑scale training pipelines.
  • Back‑ending a VectorDB that needs sub‑millisecond vector look‑ups.
  • Enabling agentic systems that store and retrieve huge conversation histories in real time.

Getting started

  1. Install the Python connector (for vLLM integration):
    pip install openlake-vllm
openlaked   # starts a local OpenLake daemon
  2. Run vLLM with the connector:
    export PYTHONHASHSEED=0
vllm serve <model_name> \
  --kv-transfer-config '{"kv_connector":"OpenLakeConnector","kv_connector_module_path":"openlake_client.openlake_connector","kv_role":"kv_both","kv_connector_extra_config":{"openlake_nodes":["127.0.0.1:9400"],"openlake_device":"local"}}'
    For multi‑node clusters, start openlaked on each GPU host (optionally with an RDMA‑specific TOML config) and point vLLM to the list of node addresses.
  3. Or use it as an S3‑compatible object store:
    # Build from source (Ubuntu example)
sudo apt-get install -y build-essential pkg-config clang cmake libhwloc-dev libudev-dev curl git awscli
curl -sSf https://sh.rustup.rs | sh -s -- -y && . $HOME/.cargo/env
git clone https://github.com/openlake-project/openlake.git && cd openlake
cargo build --release --bin openlaked

mkdir -p data/d0 data/d1 data/d2 data/d3
./target/release/openlaked --config crates/openlake_server/configs/storage-tcp-local.toml

# Talk to it with any S3 client
export AWS_ACCESS_KEY_ID=openlakeadmin
export AWS_SECRET_ACCESS_KEY=openlakeadmin
export AWS_DEFAULT_REGION=us-east-1
aws --endpoint-url http://127.0.0.1:9000 s3 mb s3://demo
aws --endpoint-url http://127.0.0.1:9000 s3 cp ./checkpoint.safetensors s3://demo/

Architecture snapshot

  • Core runtime – One pinned compio async runtime per physical CPU core, using Linux io_uring for low‑overhead I/O.
  • Zero‑copy path – GPUDirect Storage + RDMA lets data travel directly from NVMe or NIC to GPU memory.
  • Burst‑aware networking – PacedRDMA throttles senders during traffic spikes to protect latency.
  • Durability – SIMD Reed‑Solomon erasure coding spreads data across drives, offering fault tolerance with minimal space overhead.

Community & support

License

  • Apache License 2.0 (permissive, commercial‑friendly).

OpenLake is positioned as the storage layer that keeps modern GPU farms fed, reducing idle time and cost for both inference and training of large language models.

Related

  • Dispatch
    OpenLake MLPerf Storage v3.0 Performance Results

    OpenLake achieved the highest read and write bandwidth for Llama 3.1 8B checkpointing in the MLPerf Storage v3.0 Closed division S3 results, reaching 6.72 GiB/s write and 11.55 GiB/s read bandwidth.

  • Project
    bytedance/InfiniStore

    InfiniStore is an open‑source, high‑performance key‑value store that lets LLM inference clusters share and reuse KV attention caches across nodes, reducing GPU memory pressure and latency. It works with vLLM today, supports TCP and RDMA networks, and can also be used as a generic KV store for other LLM services.

  • Project
    ModelTC/LightLLM

    LightLLM is a Python‑based, high‑performance inference and serving framework for large language models. It combines fast kernels (FlashAttention, Triton) with lightweight, token‑level KV‑cache management, supports multi‑GPU scaling, and adds research‑grade features like constrained decoding and SLA‑aware request scheduling.

  • Project
    vllm-project/vllm

    vLLM is an open‑source library for high‑throughput, low‑latency serving of large language models. It introduces PagedAttention for efficient KV‑cache handling, supports many quantisation formats, integrates fast attention kernels, and offers an OpenAI‑compatible API that runs on GPUs, CPUs and other accelerators.

  • Dispatch
    Fast & Efficient LLM Inference with vLLM Course Launch

    vLLM, in collaboration with DeepLearning.AI and Red Hat, has launched a free intermediate course titled Fast & Efficient LLM Inference with vLLM to teach the full AI deployment lifecycle.