mirage-project/mirage

Mirage Persistent Kernel: Compiling LLMs into a MegaKernel

What it solves

Mirage Persistent Kernel (MPK) addresses the high latency associated with multi-GPU LLM inference. It eliminates the overhead of launching multiple GPU kernels and the communication bottlenecks between them by fusing the entire inference process into a single "megakernel."

How it works

MPK acts as a compiler and runtime system that transforms the LLM's computation graph into a single fused GPU kernel. Instead of launching separate kernels for each operation, it performs all necessary computation and communication within one launch. Developers define the kernel's inputs, outputs, and the computation graph by chaining fused operations (such as rmsnorm_linear_layer) and then compile the graph into an optimized executable kernel.

Who it’s for

ML engineers and researchers focusing on low-latency LLM inference and GPU optimization who want to reduce inference latency without requiring extensive manual CUDA programming.

Highlights

  • Significant Latency Reduction: Reduces LLM inference latency by 1.2x to 6.7x.
  • End-to-End Fusion: Fuses computation and communication into a single GPU kernel launch.
  • Automatic Compilation: Allows compiling Hugging Face models into megakernels using a high-level Python API.
  • Integrated Profiling: Includes tools to visualize the execution timeline of each task within the megakernel.

Related

  • Project
    microsoft/BitNet

    An official inference framework for 1-bit (ternary) LLMs that enables fast, energy-efficient, and lossless inference on CPUs and GPUs.

  • Project
    flashinfer-ai/flashinfer

    FlashInfer is a Python library that supplies ultra‑fast, hardware‑aware CUDA kernels for LLM inference—covering attention, GEMM, MoE, sampling, and related ops. It automatically selects the best backend (FlashAttention‑2/3, cuDNN, CUTLASS, TensorRT‑LLM), supports low‑precision FP8/FP4, works with all modern NVIDIA GPUs, and is already used in major serving frameworks like vLLM and TensorRT‑LLM.

  • Project
    deepseek-ai/DeepGEMM

    A high-performance CUDA kernel library for LLMs that provides optimized GEMM and fused MoE primitives for NVIDIA SM90 and SM100 GPUs.

  • Project
    InternLM/lmdeploy

    A toolkit for compressing, deploying, and serving LLMs and VLMs, offering high-performance inference engines and quantization to maximize throughput.

  • Project
    Mesh-LLM/mesh-llm

    A distributed LLM serving system that pools GPUs and memory across multiple machines to run large models via an OpenAI-compatible API.