awslabs/awsome-distributed-ai

Collection of best practices, reference architectures, model training examples and utilities to train large models on AWS.

What it solves

This project provides a comprehensive set of reference architectures and test cases to simplify the deployment and optimization of distributed machine learning training on AWS. It helps users avoid the complexity of manually configuring high-performance compute clusters, networking, and orchestration for large-scale models.

How it works

The repository is organized into several functional modules:

  • Architectures: Provides CloudFormation templates to deploy compute environments using services like SageMaker HyperPod, AWS ParallelCluster, AWS Batch, and Amazon EKS.
  • Custom Images: Uses Packer and Ansible to create optimized Amazon Machine Images (AMIs) and containers for training.
  • Test Cases: Offers pre-configured examples for various frameworks (PyTorch, Megatron-LM, JAX, NeMo) and parallelization techniques (DDP, FSDP) to validate model training across different sizes.
  • Validation & Observability: Includes tools for GPU cluster health checks, PyTorch environment validation, and monitoring via Prometheus and Grafana.
  • Micro-benchmarks: Provides performance tests for network communication (NCCL, NCCOM, NVSHMEM) and expert parallelism (MoE).

Who it’s for

ML engineers and infrastructure architects who need to set up, scale, and optimize distributed training workloads for large models on AWS infrastructure.

Highlights

  • Multi-service support: Templates for HyperPod, EKS, ParallelCluster, and AWS Batch.
  • Framework agnostic: Includes test cases for PyTorch, JAX, and Megatron-LM.
  • Performance focused: Dedicated tools for EFA monitoring, GPU health checks, and communication micro-benchmarks.
  • End-to-end guidance: Includes workshops and blog posts for operationalizing LLM training clusters.