fluid-cloudnative/fluid

What it solves

Fluid addresses the performance bottlenecks associated with data access in data-intensive applications, specifically for big data and AI training jobs running on Kubernetes. It provides a way to accelerate data access by managing and caching datasets across a distributed system, reducing the time computing engines (like TensorFlow or Spark) need to wait for data from remote storage.

How it works

Fluid acts as a Kubernetes-native Distributed Dataset Orchestrator and Accelerator. It introduces two key concepts:

• Dataset: A logical grouping of related data used by computing engines. Fluid provides a unified abstraction for these datasets regardless of the storage source.
• Runtime: The component that enforces isolation, sharing, and version management, and provides the interfaces to handle datasets throughout their lifecycle to enable data acceleration.

The system combines data caching technology with elastic scaling and data affinity-scheduling to ensure that data is physically closer to the computing tasks, ensuring high performance.

Who it’s for

Developers and platform engineers managing AI and big data workloads on Kubernetes who need to optimize data loading speeds and data access performance for training jobs or analytics.

Highlights

• Dataset Abstraction: Unified abstraction for datasets from multiple storage sources with observability features.
• Scalable Cache Runtime: A unified access interface for data operations across different runtimes and third-party storage systems.
• Automated Data Operations: Support for various automated operation modes to facilitate system integration.
• Elasticity and Scheduling: Combines caching with elastic scaling and data affinity-scheduling to improve performance.
• Platform Agnostic: Supports native, edge, Serverless Kubernetes clusters, and multi-cluster environments.