ray-project/raydp

What it solves

RayDP eliminates the operational complexity and latency associated with running separate clusters for big data processing (Spark) and AI model training (PyTorch, TensorFlow). It allows developers to build end-to-end data and AI pipelines within a single Python program, removing the need for external glue code or complex workflow orchestrators like AirFlow or KubeFlow.

How it works

RayDP enables Spark to run on top of Ray by using Ray as the Spark resource manager and running Spark executors as Ray actors. It provides APIs to convert Spark DataFrames into Ray Datasets, allowing data to be exchanged in-memory for high performance. Additionally, it offers scikit-learn style Estimator APIs (TorchEstimator and TFEstimator) that wrap Ray Train to simplify distributed training directly on Spark DataFrames.

Who it’s for

• ML Infrastructure Teams: To build unified ML platforms that integrate Spark and other AI components on a single substrate.
• Data Scientists: To write PySpark code alongside AI libraries and scale seamlessly from a laptop to the cloud.
• Data Engineers: To run on-demand Spark jobs in the cloud with auto-scaling without manually setting up a Spark cluster.

Highlights

• Unified Execution: Runs Spark executors within Ray actors, consolidating data processing and AI training in one environment.
• In-Memory Data Exchange: High-performance conversion between Spark DataFrames and Ray Datasets.
• Simplified Training: High-level Estimator APIs for distributed PyTorch and TensorFlow training on Spark data.
• Fault Tolerance: Supports recoverable pipelines for converting Spark DataFrames to Ray Datasets, ensuring resilience against executor loss.