awslabs/awsome-distributed-ai

Best practices, reference architectures, and examples for distributed AI training and inference on AWS.

What it solves

This project provides a comprehensive set of reference architectures and runnable examples to simplify the deployment and operation of distributed AI training and inference on AWS infrastructure. It addresses the complexity of setting up high-performance GPU clusters, configuring network optimizations (like EFA), and implementing distributed training frameworks across different AWS services.

How it works

The repository is organized into several functional modules:

  • Architectures: Deployable templates (CloudFormation, Terraform) for clusters using SageMaker HyperPod, AWS ParallelCluster, AWS PCS, and Amazon EKS.
  • Examples: Framework-centric guides for training (using PyTorch DDP/FSDP, Megatron-LM, NeMo) and inference (using vLLM, SGLang, NVIDIA Dynamo), as well as use-case-specific end-to-end demos.
  • Custom AMIs: Scripts using Packer and Ansible to build optimized machine images.
  • Validation & Observability: Tools for checking GPU cluster health, validating PyTorch environments, and monitoring network performance via Prometheus and Grafana.
  • Micro-benchmarks: Tools to evaluate communication performance using NCCL, NCCOM, and NVSHMEM.

Who it’s for

It is designed for ML engineers and infrastructure architects who need to scale AI workloads to multiple GPUs and multiple nodes on AWS, specifically those working with large-scale model training and high-throughput inference.

Highlights

  • Multi-Service Support: Covers SageMaker HyperPod, ParallelCluster, PCS, and EKS.
  • Framework Agnostic: Includes examples for a wide range of distributed training and serving engines.
  • Infrastructure as Code: Provides ready-to-use templates for rapid cluster deployment.
  • Health & Performance Tools: Includes built-in validation and micro-benchmarking tools to ensure cluster stability and performance.

Related

  • Project
    Arm-Examples/ML-examples

    A collection of machine learning examples and tutorials for deploying and optimizing AI models on Arm Ethos NPUs and Cortex-based platforms.

  • Dispatch
    Building Blocks for Foundation Model Training and Inference on AWS

    Hugging Face details a four-layer architectural framework on AWS—comprising infrastructure, resource orchestration, ML software stacks, and observability—to support the evolving scaling laws of pre-training, post-training, and test-time compute.

  • Project
    CerebriumAI/examples

    A collection of ready-to-use deployment examples for the Cerebrium platform, covering LLMs, voice agents, image generation, and inference optimization.

  • Project
    aws/deep-learning-containers

    AWS Deep Learning Containers provide pre‑built, security‑patched Docker images for popular AI/ML frameworks (PyTorch, TensorFlow, vLLM, Ray, SGLang, etc.) that run directly on AWS services like EC2, SageMaker, and EKS, simplifying setup, version management, and large‑model serving.

  • Project
    EvolvingLMMs-Lab/lmms-engine

    A unified, high-performance training engine for multimodal models that integrates distributed training, kernel fusion, and advanced optimizers to enable scalable pretraining and fine-tuning.