OpenLake MLPerf Storage v3.0 Performance Results
OpenLake has achieved the highest read and write bandwidth among comparable Closed division S3 submissions in the MLPerf Storage v3.0 benchmark for Llama 3.1 8B checkpointing. This result demonstrates the system's ability to minimize GPU idle time during state saving and accelerate recovery after failures in large-scale LLM training.
MLPerf Storage v3.0 Benchmark Results
OpenLake led the comparable S3 submissions for the Llama 3.1 8B configuration, which emulates the Llama 3 family of models. The benchmark tests a storage system's behavior under full load when training processes save or restore model state.
Performance Metrics
For the Llama 3.1 8B configuration, OpenLake delivered the following performance metrics:
| Metric | OpenLake Result |
|---|---|
| Write Bandwidth | 6.72 GiB/s |
| Write Duration | 29.42 seconds |
| Read Bandwidth | 11.55 GiB/s |
| Read Duration | 9.37 seconds |
Comparison with Other Submissions
When compared to other published results using the same Closed division checkpointing workload, S3 API interface, Llama 3.1 8B model, one client node, and one data parallel instance, OpenLake outperformed the competitors:
| Organization | Write Bandwidth | Read Bandwidth | Write Duration | Read Duration | | :--- | :--- | :--- | :--- | | OpenLake | 6.72 GiB/s | 11.55 GiB/s | 29.42 s | 9.37 s | | NVIDIA AIStore (6 node) | 3.40 GiB/s | 11.08 GiB/s | 30.83 s | 9.47 s | | NVIDIA AIStore (12 node) | 3.20 GiB/s | 8.33 GiB/s | 30.83 s | 12.90 s | | NVIDIA AIStore (3 node) | 3.02 GiB/s | 6.99 GiB/s | 34.67 s | 15.00 s | | Nebius Object Storage | 2.81 GiB/s | 7.14 GiB/s | 37.24 s | 14.67 s |
OpenLake's write bandwidth of 6.72 GiB/s was 1.98× the next fastest comparable submission.
Technical Architecture of OpenLake
OpenLake utilizes the Infinity Core I/O Engine to achieve high-performance storage for AI workloads. The system is powered by io_uring and GPUDirect Storage, designed for low latency and high throughput.
Key Implementation Details
To maximize I/O operations in flight while reducing CPU overhead, OpenLake employs the following techniques:
- Asynchronous io_uring I/O: Used for nonblocking kernel operations.
- Pinned Execution Threads: Reduces scheduling overhead.
- Fine-grained I/O Coalescing: Optimizes data movement.
- XFS and Workload-Specific Tuning: Tailored for AI storage requirements.
- S3 API Interface: Provides a standard interface for reading and writing checkpoints.
Additionally, the project uses a Rust compio (thread-per-core design) with on-GPU compression to support KV offloading, achieving over 1 million IOPS in 1ms on a 96-node cluster.
The Impact of Checkpoint Performance on LLM Training
Checkpointing is the process of periodically saving model weights, optimizer state, and training state to storage to protect progress in large-scale training runs that can last for weeks.
Reducing Training Costs
In synchronous checkpointing, training pauses until the state is written durably. Faster write bandwidth reduces the time GPUs sit idle, which improves accelerator utilization and lowers overall training costs.
Accelerating Recovery
Faster read bandwidth reduces the recovery path after a failure, allowing training to resume more quickly. This is critical for reinforcement learning, pretraining, and post-training jobs where checkpoints are created frequently and infrastructure failures can occur.