feifeibear/long-context-attention

What it solves

YunChang implements Unified Sequence Parallelism (USP), which addresses the limitations of two common distributed attention methods used for long-context LLM training and inference: DeepSpeed-Ulysses and Ring-Attention. Ulysses is limited by the number of attention heads (making it unsuitable for GQA or MQA), while Ring-Attention is often less efficient in computation and communication and can risk deadlocks in large-scale deployments.

How it works

USP acts as a hybrid sequence parallel approach that combines the strengths of both DeepSpeed-Ulysses and Ring-Attention. It allows users to set both a Ulysses degree and a Ring degree to partition the sequence across GPUs. The project integrates with various attention backends, including FlashAttention (v2 and v3), Torch Efficient, and others like SageAttention and FlashInfer. It also provides load-balancing strategies (such as "zigzag" and "stripe") for causal masks to optimize performance.

Who it’s for

It is designed for researchers and engineers training or deploying long-context generative AI models who need to scale sequence lengths across multiple GPUs, particularly those working with GQA/MQA architectures or heterogeneous network hardware.

Highlights

• Hybrid Approach: Synergizes Ulysses and Ring attention to overcome head-count limitations and communication inefficiencies.
• Broad Hardware Support: Compatible with NVIDIA GPUs (H100, A100, etc.) and NPUs.
• Flexible Backends: Supports FlashAttention v2/v3, SageAttention, and FlashInfer.
• Integration: Already applied in NVIDIA's TransformerEngine and compatible with Megatron-LM frameworks.