openlake-project/openlake

What it solves

OpenLake is a high-performance distributed storage engine designed specifically for GPU workloads. It addresses the bottleneck of slow storage I/O, which often leaves GPUs idle during training and inference. By providing ultra-low latency and high throughput, it ensures that accelerators are fully utilized.

How it works

Built with Rust and utilizing Linux io_uring, OpenLake minimizes the path from storage to GPU memory. It employs several key technologies:

• Zero Copy: Uses GPUDirect Storage and RDMA to move data directly between NVMe/NICs and GPU memory, bypassing the host CPU and page cache.
• Core Local Asynchronous I/O: Runs a pinned runtime per physical core to avoid cross-core contention.
• Burst Aware RDMA: Implements credit-based flow control to prevent receiver overwhelm and protect tail latency.
• Erasure Coding: Uses SIMD Reed Solomon erasure coding for durable, capacity-efficient storage.
• KV Cache Offload: Allows LLM inference engines (like vLLM) to offload and retrieve KV caches at petabyte scale across a GPU fleet.

Who it’s for

It is intended for AI infrastructure engineers and ML practitioners who need to optimize LLM inference costs, speed up model training, and manage massive context storage for AI agents.

Highlights

• Extreme Performance: Delivers million+ IOPS within 1ms and has topped MLPerf Storage v3.0 object checkpointing benchmarks.
• S3 Compatibility: Provides an S3-compatible object store for GPU fleets.
• KV Cache Management: Significantly reduces time to first token (TTFT) for long and repeated prompts by caching prefixes across GPU nodes.
• Lossless Compression: Includes ExANS, a lossless GPU codec for BF16 KV cache to reduce costs.
• Broad Utility: Supports fast index building for VectorDBs and ultra-fast checkpointing for RL/ML workloads.