cubefs/cubefs

cloud-native distributed storage

What it solves

CubeFS provides a scalable, cloud-native distributed storage solution that eliminates the tight coupling between compute and storage. This allows databases, search systems, and AI/ML applications to scale their processing power and storage capacity independently.

How it works

It functions as a distributed file and object storage system that supports multiple access protocols (POSIX, HDFS, S3, and a native REST API). It utilizes a highly scalable metadata service with strong consistency and offers flexible storage policies, including high-performance replication and low-cost erasure coding. For hybrid cloud setups, it provides I/O acceleration via multi-level caching on top of public cloud storage like S3.

Who it’s for

It is designed for developers and infrastructure engineers building data lakes, datacenter filesystems, or private/hybrid cloud storage, specifically those running AI/ML workloads or large-scale container platforms.

Highlights

  • Supports POSIX, HDFS, and S3 protocols for flexible data access.
  • Enables separation of storage and compute for AI/ML and database architectures.
  • Features multi-tenancy for better resource isolation and utilization.
  • Optimized for both large and small files, as well as sequential and random writes.
  • Provides hybrid cloud acceleration through multi-level caching.

Related

  • Project
    kubeflow/kubeflow

    Kubeflow is a Kubernetes-native foundation of tools for building AI platforms, providing a modular ecosystem to manage the entire AI lifecycle.

  • Project
    fluid-cloudnative/fluid

    A Kubernetes-native distributed dataset orchestrator and accelerator that optimizes data access for AI and big data applications by providing dataset abstraction and elastic caching.

  • Project
    data-infra/cube-studio

    A cloud-native, one-stop MLOps platform that manages the entire AI lifecycle from data labeling and distributed training to model deployment, with extensive support for heterogeneous domestic AI hardware.

  • Project
    kubeflow/trainer

    A Kubernetes-native distributed AI platform for scalable LLM fine-tuning and training across various frameworks like PyTorch, JAX, and XGBoost.

  • Project
    kubesphere/kubesphere

    KubeSphere is a distributed operating system for cloud-native application management that simplifies Kubernetes management across multi-cloud, edge, and datacenter environments.