microsoft/MInference

[NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.

What it solves

MInference addresses the high computational cost and slow speed of the pre-filling stage in long-context Large Language Models (LLMs). Processing million-token prompts typically requires significant time and memory, which MInference aims to accelerate while maintaining model accuracy.

How it works

The project leverages the observation that LLM attention is dynamically sparse and often follows static patterns. It works by:

  1. Offline Analysis: Determining which sparse pattern each attention head belongs to before runtime.
  2. Online Approximation: Approximating the sparse index during inference.
  3. Custom Kernels: Using optimized custom kernels to compute attention dynamically based on these patterns.

Additionally, the project includes MMInference for vision-language models (VLMs), which uses modality-aware permutation sparse attention to handle grid patterns in visual inputs and modality boundaries.

Who it’s for

Developers and researchers working with long-context LLMs (such as LLaMA-3, Qwen2.5, and GLM-4) who need to reduce pre-filling latency and memory usage on hardware like A100 GPUs.

Highlights

  • Significant Speedup: Achieves up to 10x faster pre-filling for 1M context on a single A100.
  • Broad Integration: Integrated into Qwen2.5-1M and supported by inference engines like vLLM and SGLang.
  • Wide Model Support: Compatible with various LLaMA-style, Phi, and other open-source long-context LLMs.
  • Comprehensive Tooling: Includes SCBench, a KV cache-centric benchmark for evaluating long-context methods across generation, compression, retrieval, and loading.