SandAI-org/MagiCompiler

What it solves

MagiCompiler addresses the memory walls and operator overheads that limit the performance of large-scale Transformer-like architectures. It moves beyond simple local operator optimization to provide system-level optimizations for both training and multi-modality inference.

How it works

Built on top of torch.compile, MagiCompiler acts as a global manager for the execution lifecycle. It manages subgraph dispatching, orchestrates dataflow (including offloading and prefetching), and controls memory allocation. Key technical mechanisms include:

• Full-graph capture: Maximizes kernel fusion scope across Transformer boundaries for inference.
• FSDP-aware layer-wise compilation: Enables aggressive cross-op fusion during training while keeping distributed parameter sharding transparent.
• Selective offloading: Overlaps host-to-device (H2D) transfers with computation to eliminate pipeline bubbles in memory-constrained environments.
• Heuristic activation recomputation: Automatically determines which operations to save and which to recompute based on whether they are compute-bound or memory-bound to reduce peak memory.

Who it’s for

It is designed for developers and researchers working with large-scale Transformer models, particularly those implementing multi-modality frameworks or training large models using distributed strategies like FSDP.

Highlights

• Plug-and-play integration: Uses simple decorators (@magi_compile and @magi_register_custom_op) to achieve speedups without requiring complex model refactoring.
• Significant acceleration: Demonstrates 9% to 26% performance gains over SOTA solutions on NVIDIA H100 GPUs for video generation models.
• Introspection toolkit: Includes magi_depyf for debugging compilation timelines, bytecode flows, and split subgraphs.
• Hardware-aware: Supports high-end GPUs (H100) and consumer GPUs (RTX 5090) through JIT offloading.