airllm: a memory-efficient inference engine that runs massive LLMs on low-end GPUs by loading one layer at a time

What it solves

AirLLM allows users to run massive large language models (LLMs) on consumer-grade hardware with very limited VRAM. It enables the execution of models like Llama 3.1 405B or DeepSeek-V3 671B on GPUs with as little as 8GB to 12GB of memory, bypassing the need for expensive high-end GPUs or complex model compression techniques like distillation or pruning.

How it works

Instead of loading the entire model into VRAM, AirLLM decomposes the model and saves it layer-wise. During inference, it only keeps one layer of the model on the GPU at a time, meaning the VRAM requirement is determined by the size of a single layer rather than the total model size. It also supports optional block-wise quantization (4-bit or 8-bit) to reduce the size of weights on disk and speed up loading times by up to 3x.

Who it’s for

Developers and researchers who want to run state-of-the-art, massive LLMs on low-end commodity computers or hobbyist GPU cards without sacrificing model accuracy.

Highlights

• Extreme VRAM Efficiency: Run 70B models on 4GB GPUs and 671B models on 12GB GPUs.
• Broad Model Support: Compatible with Llama, Qwen, DeepSeek, Mistral, Mixtral, Phi, Gemma, and others via a single AutoModel interface.
• No Accuracy Loss: Operates without requiring quantization, distillation, or pruning by default.
• Cros-Platform Support: Works on Linux and Apple Silicon (MacOS).
• Performance Optimization: Includes prefetching to overlap model loading and computation for faster inference.