awslabs/awsome-distributed-ai

What it solves

This repository provides a comprehensive set of reference architectures and runnable examples to help users deploy and operate distributed AI training and inference workloads on AWS infrastructure. It simplifies the process of setting up complex GPU clusters, configuring network optimizations, and choosing the right training or inference engine for large-scale models.

How it works

The project is organized into several functional modules:

• Architectures: Provides CloudFormation and Terraform templates to deploy compute clusters using SageMaker HyperPod, AWS ParallelCluster, AWS Parallel Computing Service (PCS), and Amazon EKS.
• Examples: Contains framework-centric examples (e.g., PyTorch DDP/FSDP, Megatron-LM, NeMo) and use-case-centric demos for both training and inference (using engines like vLLM and SGLang).
• Custom AMIs: Uses Packer and Ansible to create optimized Amazon Machine Images for these environments.
• Validation & Observability: Includes tools for GPU cluster health checks, PyTorch environment validation, and monitoring stacks using Prometheus and Grafana.
• Micro-benchmarks: Provides tools to evaluate network and communication performance via NCCL, NCCOM, and NVSHMEM.

Who it’s for

It is designed for ML engineers and infrastructure architects who need to scale AI workloads across multiple GPUs and nodes on AWS, specifically those working with large language models (LLMs) and other frontier AI models.

Highlights

• Multi-Compute Support: Ready-to-use templates for HyperPod, EKS, ParallelCluster, and PCS.
• Framework Diversity: Support for major distributed training frameworks like Megatron-LM, NeMo, and FSDP.
• Inference Optimization: Examples for high-performance serving engines such as vLLM and SGLang.
• Cluster Health Tools: Integrated validation and observability tools to ensure hardware and network stability.