SandAI-org/MagiCompiler

What it solves

MagiCompiler addresses memory bottlenecks and operator overheads in large-scale Transformer-like architectures. It moves beyond simple local operator optimization to provide system-level orchestration of dataflow and memory, reducing the "memory wall" effect during both training and multi-modality inference.

How it works

Built on top of torch.compile, MagiCompiler acts as a global manager for the execution lifecycle. It uses full-graph capture for inference to maximize kernel fusion and FSDP-aware layer-wise compilation for training to enable cross-op fusion while keeping distributed parameter sharding transparent. It also implements a selective offloading policy to overlap H2D transfers with computation and a heuristic activation recomputation strategy that automatically determines which operations to save or recompute based on whether they are compute-bound or memory-bound.

Who it’s for

It is designed for developers and researchers working with large-scale Transformer models, particularly those implementing multi-modality frameworks or training large models in distributed environments.

Highlights

• Unified Framework: Supports both inference (full-graph capture) and training (FSDP-aware compilation).
• Plug-and-Play: Uses simple decorators (@magi_compile and @magi_register_custom_op) to integrate into existing models with minimal code changes.
• Smart Offloading: Built-in asynchronous offloading to eliminate pipeline bubbles in memory-constrained setups.
• Automated Recomputation: Replaces manual checkpointing with heuristic-based activation recomputation to lower peak memory.
• Introspection Tools: Includes magi_depyf for debugging compilation timelines and decompiled bytecode flows.