CurvineIO/curvine

AI-Native & Cloud-Native FS: A high-performance file semantic layer for cloud object storage, integrated with high-speed cache. CNCF Sandbox Project.

What it solves

Curvine is a high-performance file system designed to bridge the gap between cloud object storage (like AWS S3) and the high-speed data access requirements of large-scale AI workloads. It solves the problem of slow data retrieval and high latency when AI training, inference, and AI Agent platforms need to access massive datasets or model checkpoints stored in the cloud.

How it works

Curvine layers a distributed POSIX-compliant file system over cloud object storage. It uses a multi-tier distributed cache (Memory $\rightarrow$ SSD $\rightarrow$ HDD) to keep hot data close to the compute nodes.

Its architecture consists of:

  • Access Layer: Supports AI Agent pods and big-data engines.
  • Protocol Layer: Provides multiple access paths including a FUSE client, S3-compatible gateway, HDFS adapter, and a Kubernetes CSI driver.
  • Cluster Core: A Raft-replicated Master node manages metadata and coordination, while Worker nodes handle the actual data I/O and caching.
  • Storage Layer: Uses multi-cloud object storage as the durable persistence layer.

Who it’s for

  • AI Engineers: Those accelerating LLM training and model distribution across multiple regions.
  • AI Agent Platform Developers: Those needing to provide isolated, high-performance POSIX workspaces for thousands of stateful Agent pods on Kubernetes.
  • Data Engineers: Those managing multimodal data lakes or accelerating OLAP queries using compute-storage separated architectures.

Highlights

  • AI-Native Design: Specifically optimized for AI training acceleration and AI Agent storage.
  • Extreme Performance: Built in Rust with a zero-copy data path, achieving ~100$\mu$s-class latency and 100K+ QPS.
  • Massive Metadata Capacity: Supports up to 5 billion small files per cluster, making it suitable for high-concurrency Agent workloads.
  • Cloud-Native Integration: Native Kubernetes CSI driver for millisecond-level PVC provisioning.
  • Multi-Cloud Compatibility: Works transparently across AWS S3, Azure Blob, Google GCS, and other S3-compatible stores.

Related

  • Project
    fluid-cloudnative/fluid

    A Kubernetes-native distributed dataset orchestrator and accelerator that optimizes data access for AI and big data applications by providing dataset abstraction and elastic caching.

  • Project
    cubefs/cubefs

    CubeFS is a CNCF graduated distributed file and object storage system that enables the separation of storage and compute for AI/ML applications and data lakes.

  • Project
    NVIDIA/aistore

    A high-performance distributed storage stack tailored for AI workloads, providing linear scalability and optimized data retrieval for ML pipelines across multi-cloud environments.

  • Project
    Fergana-Labs/stash

    A persistent knowledge base and shared memory layer for AI agents that connects to various data sources and records session transcripts to maintain context across runs.

  • Project
    activeloopai/hivemind

    A shared, cloud-backed memory system for AI coding agents that captures session traces and codifies them into reusable skills shared across a team.