intelligent-machine-learning/dlrover

What it solves

DLRover addresses the engineering complexities of distributed deep learning training for large AI models. It specifically targets the instability, downtime, and resource inefficiency associated with training on large clusters (K8s/Ray), allowing developers to focus on model architecture rather than hardware acceleration or distributed runtime management.

How it works

DLRover provides an automated operation and maintenance layer for training jobs. It employs several key mechanisms:

• Fault Tolerance: It automatically diagnoses failures and restarts either the specific process (for software errors) or the failed node (for hardware errors) without stopping the entire job.
• Flash Checkpoint: It uses in-memory checkpoints to save and load model states in seconds, significantly reducing the time spent on I/O compared to SSD or NAS storage.
• Auto-Scaling: It monitors node workload and throughput to dynamically scale resources up or down, mitigating bottlenecks like node stragglers or unbalanced parameter server workloads.
• Dynamic Data Sharding: It splits datasets into small shards, allowing the system to recover lost shards from failed workers and reassign shards to faster workers to reduce stragglers.

Who it’s for

Model developers and ML engineers training large-scale AI models (such as LLMs) on distributed clusters using PyTorch or TensorFlow, particularly those operating on Kubernetes or Ray.

Highlights

• High Goodput: Increased the goodput of GLM-65B training on thousands of GPUs from 69% to 95%.
• Rapid Recovery: Flash Checkpoint allows recovery from failures in seconds via shared memory.
• Broad Framework Support: Provides integration for PyTorch (DDP, FSDP, DeepSpeed, Megatron-LM) and TensorFlow.
• Infrastructure Flexibility: Supports both Kubernetes and Ray architectures.
• Online Learning: Supports real-time streaming data integration via message queues like Kafka or Pulsar.