flexflow/flexflow-train

Automatically Discovering Fast Parallelization Strategies for Distributed Deep Neural Network Training

What it solves

FlexFlow Train addresses the challenge of optimizing distributed deep neural network (DNN) training. Instead of requiring developers to manually configure complex parallelization strategies, it automatically searches for the most efficient way to distribute the workload across hardware.

How it works

The framework uses an autotuning process to discover optimal parallelization strategies. It can import models from PyTorch, TensorFlow Keras, and ONNX, and then search for the best distribution of data and model parameters. It supports multiple types of parallelism, including data, model, parameter, and attribute parallelism, and provides both Python and C++ interfaces for implementation.

Who it’s for

It is designed for deep learning researchers and engineers who need to scale their training across multiple GPUs or nodes and want to maximize hardware performance without manually tuning parallelization settings.

Highlights

  • Automatic Parallelization Search: Automatically finds efficient strategies to accelerate distributed training.
  • Broad Compatibility: Supports importing models from PyTorch, TensorFlow Keras, and ONNX.
  • Flexible Parallelism: Explores data, model, parameter, and attribute parallelism.
  • Multi-Language Support: Offers both Python and C++ APIs for development.

Related

  • Project
    hpcaitech/ColossalAI

    A distributed deep learning framework that makes training and inference for large AI models faster and cheaper through advanced parallelism and memory management.

  • Project
    rwth-i6/returnn

    An extensible training framework for universal recurrent neural networks optimized for fast multi-GPU training and decoding.

  • Project
    microsoft/onnxruntime

    A cross-platform machine learning accelerator that optimizes inference and training for models from various frameworks like PyTorch and TensorFlow.

  • Project
    uxlfoundation/oneDNN

    A cross-platform performance library providing optimized building blocks for deep learning applications to improve performance across various CPUs and GPUs.

  • Project
    intelligent-machine-learning/dlrover

    An automatic distributed deep learning system that simplifies large-scale AI model training by providing automated fault tolerance, auto-scaling, and rapid checkpoint recovery on K8s and Ray.