lyogavin/airllm

AirLLM 70B inference with single 4GB GPU

What it solves

AirLLM allows users to run massive large language models (LLMs) on consumer-grade hardware with very limited VRAM. It eliminates the need for quantization, distillation, or pruning to fit huge models—such as Llama 3.1 405B or DeepSeek-V3 671B—onto single GPUs with as little as 4GB to 12GB of memory.

How it works

Instead of loading the entire model into GPU memory, AirLLM decomposes the model and streams it layer-by-layer. By keeping only one layer on the GPU at a time, the VRAM requirement is determined by the size of a single layer rather than the total model size. For sparse Mixture-of-Experts (MoE) models, it further optimizes this by streaming only the experts that a token actually routes to.

Who it’s for

Developers and researchers who want to run state-of-the-art, high-parameter LLMs on low-end commodity computers or single hobbyist GPU cards without sacrificing model precision.

Highlights

  • Extreme Memory Efficiency: Run 70B models on 4GB VRAM and 671B models on ~12GB VRAM.
  • Broad Model Support: Compatible with Llama, Qwen, DeepSeek, Mistral, Mixtral, Phi, Gemma, and Kimi K3.
  • Wide Hardware Support: Works on Linux, Windows, and Apple Silicon (MacOS).
  • Optional Speed-up: Supports block-wise quantization (4-bit or 8-bit) to reduce disk loading bottlenecks and increase inference speed by up to 3x.
  • Simple Integration: Uses an AutoModel class that mimics the Hugging Face transformers API for easy adoption.

Related

  • Dispatch
    AirLLM enables 70B LLM inference on a single 4GB GPU

    AirLLM lets you run 70‑billion‑parameter LLMs on a single 4 GB GPU by streaming one layer at a time, without quantization, as demonstrated in the lyogavin/airllm repository.

  • Project
    turboderp-org/exllamav3

    An inference library for running local LLMs on consumer GPUs featuring EXL3 quantization, parallel inference, and CPU offloading for large MoE models.

  • Project
    ModelTC/LightLLM

    LightLLM is a Python‑based, high‑performance inference and serving framework for large language models. It combines fast kernels (FlashAttention, Triton) with lightweight, token‑level KV‑cache management, supports multi‑GPU scaling, and adds research‑grade features like constrained decoding and SLA‑aware request scheduling.

  • Project
    ggml-org/llama.cpp

    A plain C/C++ implementation for high-performance LLM and VLM inference across a wide range of hardware with minimal setup.

  • Project
    mlc-ai/web-llm

    A high-performance in-browser LLM inference engine that uses WebGPU to run language models locally without a server.