mirage-project/mirage

Mirage Persistent Kernel: Compiling LLMs into a MegaKernel

What it solves

Mirage Persistent Kernel (MPK) addresses the latency issues associated with multi-GPU LLM inference. It reduces inference latency by 1.2x to 6.7x by eliminating the overhead of multiple kernel launches and communication between GPUs.

How it works

MPK is a compiler and runtime system that transforms LLM inference into a single "megakernel"—a fused GPU kernel that handles all computation and communication within a single launch. Users define the computation graph by chaining fused operations (such as combining RMSNorm and Linear layers) and specifying tensor allocations. The system then compiles this graph into a high-performance, persistent kernel that executes on the GPU.

Who it’s for

Developers and researchers working on LLM inference optimization, specifically those looking to reduce latency in multi-GPU environments using CUDA and PyTorch.

Highlights

  • End-to-end GPU fusion: Fuses all necessary computation and communication into one kernel launch.
  • Significant latency reduction: Achieves 1.2x to 6.7x speedups over native Triton and FlashInfer kernels.
  • Automatic transformation: Automatically transforms models from the Hugging Face model zoo into megakernels.
  • Integrated profiling: Includes tools to visualize the execution timeline of each task within the megakernel.

Related

  • Project
    deepseek-ai/DeepGEMM

    A high-performance CUDA kernel library for LLMs that optimizes GEMM and MoE operations, featuring runtime JIT compilation and extreme throughput on NVIDIA SM90/SM100 GPUs.

  • Project
    microsoft/BitNet

    An official inference framework for 1-bit (ternary) LLMs that enables fast, energy-efficient, and lossless inference on CPUs and GPUs.

  • Project
    lucienhuangfu/eLLM

    An LLM inference framework for CPU servers that optimizes memory usage to outperform GPUs in long-horizon, multi-turn inference and ultra-long context tasks.

  • Project
    Luce-Org/lucebox

    Lucebox is an open‑source C++ inference engine for large language models that uses speculative pre‑fill and custom GPU kernels (CUDA/HIP) to dramatically speed up token generation on consumer and mixed‑GPU setups. It supports many models (Qwen, DeepSeek, Gemma, etc.), offers vision input, and provides ready‑made hardware‑specific profiles for AMD and NVIDIA cards, delivering hundreds of tokens per second on a single GPU and even higher throughput in heterogeneous configurations.

  • Project
    InternLM/lmdeploy

    LMDeploy is a comprehensive toolkit for compressing, deploying, and serving large language and vision models, designed to maximize inference speed and hardware efficiency.