ByteDance-Seed/Triton-distributed

Distributed Compiler and Optimized Parallel Kernels

What it solves

Triton-distributed addresses the difficulty of developing highly optimized distributed kernels that overlap computation and communication. It allows programmers to create efficient kernels (such as Distributed-GEMM) that can compete with hand-tuned libraries while maintaining the flexibility of a compiler-based approach.

How it works

Based on OpenAI Triton, this distributed compiler provides a set of low-level primitives that enable the design of kernels where data movement between GPUs occurs simultaneously with computation. It supports multiple hardware backends, including Nvidia (SM80, SM89, SM90a) and AMD (CDNA3) GPUs, and utilizes communication layers like NVLink, InfiniBand (IB), and PCIe.

Who it’s for

It is designed for GPU kernel developers and AI researchers who need to optimize distributed training or inference performance for large-scale models, specifically those working with Tensor Parallelism (TP) and Expert Parallelism (EP).

Highlights

  • Compute-Communication Overlap: Specifically designed to hide communication latency by overlapping it with computation.
  • Broad Hardware Support: Compatible with both Nvidia and AMD GPUs.
  • High Performance: Achieves performance comparable to or better than hand-tuned libraries for operations like AllGather GEMM and GEMM ReduceScatter.
  • Inference Acceleration: Demonstrated speedups (e.g., 1.33x for Seed-OSS-36B-Instruct) and low-latency AllToAll implementations.

Related

  • Dispatch
    OpenAI Triton 1.0 Release

    OpenAI has released Triton 1.0, an open-source Python-like language and compiler that allows researchers to write highly efficient GPU kernels without deep CUDA expertise.

  • Project
    triton-lang/triton

    Triton is a language and compiler for writing highly efficient custom deep-learning primitives, offering higher productivity than CUDA and more flexibility than other DSLs.

  • Project
    cfregly/ai-performance-engineering

    A comprehensive collection of code and resources for optimizing AI systems, focusing on GPU tuning, distributed training, and high-throughput inference scaling.

  • Project
    flagos-ai/FlagTree

    An open-source unified compiler for multiple AI chips that leverages the Triton ecosystem to enable a "develop once, run anywhere" workflow across diverse hardware accelerators.

  • Project
    InfiniTensor/ninetoothed

    A Triton-based domain-specific language that simplifies writing high-performance GPU kernels using tensor-oriented meta-programming.