airllm: a memory-efficient inference engine that runs massive LLMs on consumer GPUs by loading one layer at a time

What it solves

AirLLM allows users to run massive large language models (LLMs) on consumer-grade hardware with very limited VRAM. It enables the execution of models as large as 671B parameters (like DeepSeek-V3) on GPUs with as little as 12GB of VRAM, or 70B models on 4GB cards, without requiring quantization, distillation, or pruning.

How it works

Instead of loading the entire model into GPU memory, AirLLM decomposes the model and saves it layer-wise. During inference, it only keeps one layer of the model on the GPU at a time, meaning the VRAM requirement is determined by the size of a single layer rather than the total model size. It also supports optional block-wise quantization (4-bit or 8-bit) to reduce the size of weights on disk and speed up loading times by up to 3x.

Who it’s for

Developers and researchers who want to run state-of-the-art, massive open-source LLMs on low-end commodity computers or hobbyist GPUs without the accuracy loss typically associated with heavy quantization.

Highlights

• Extreme VRAM Efficiency: Run 405B Llama 3.1 on 8GB VRAM or DeepSeek-V3 (671B) on 12GB VRAM.
• Broad Model Support: Works with Llama (2/3/3.1/3.3/4), Qwen, DeepSeek, Mistral, Mixtral, Phi, Gemma, ChatGLM, and more.
• No-Quantization Option: Capability to run models in full precision without needing pruning or distillation.
• Cross-Platform: Supports Linux and Apple Silicon (MacOS).
• Coded for Simplicity: Uses a single AutoModel class to automatically detect and load models from Hugging Face.