lyogavin/airllm

AirLLM – Run and Train Giant LLMs on Tiny GPUs

What it does
AirLLM is a Python library that lets you inference and fine‑tune very large language models (up to 671 B parameters) on a single consumer‑grade GPU with only a few gigabytes of VRAM. It achieves this by streaming model layers from disk and keeping only one layer (plus optional adapters) on the GPU at a time. The library also offers a lightweight block‑wise weight‑only compression (4‑bit/8‑bit) that can triple inference speed with minimal accuracy loss.

Why it matters
Normally a 70 B model needs 30‑40 GB of GPU memory. AirLLM’s layer‑wise streaming reduces the memory requirement to the size of a single layer, so a 70 B model runs on ~4 GB, a 125 B MoE on ~6 GB, and even the 671 B DeepSeek‑V3 on ~12 GB. It also supports training large models by streaming frozen weights and keeping only adapters in GPU memory, enabling fine‑tuning of 125 B models on a 6 GB card.

Key features (as described in the README)

• Layer‑wise streaming – loads one transformer layer at a time, keeping the rest on disk.
• Block‑wise weight‑only compression – 4‑bit or 8‑bit quantisation of weights only (no activation quantisation), giving up to 3× speed‑up.
• Broad model support – works with virtually any Hugging Face LLM: Llama 2/3/3.1/3.3/4, Qwen (1‑3.8, MoE, Flash‑Next, FP8, VL), DeepSeek‑V2/V3, Mixtral, Phi‑4, Gemma, ChatGLM, Baichuan, InternLM, Yi, Kimi K3, etc.
• Training on tiny VRAM – streaming frozen weights + LoRA adapters; examples for Qwen 3.8‑Flash‑Next (125 B) and Qwen 3.8‑27 B.
• Simple API – AutoModel.from_pretrained(<repo_id>, compression='4bit') works like the standard Hugging Face AutoModel.
• Cross‑platform – Linux, Windows and Apple‑silicon macOS (MLX + torch) support.
• Configurable – compression, profiling, custom shard‑saving path, Hugging Face token, prefetching, and optional deletion of the original model to save disk space.

Typical usage

from airllm import AutoModel
model = AutoModel.from_pretrained(
    "Qwen/Qwen3-32B",   # any HF model ID works
    compression='4bit'   # optional 4‑bit/8‑bit speed boost
)
# tokenise, generate, etc. – same as a regular HF model

During the first load the library splits the model into per‑layer shards on disk; subsequent runs just stream those shards.

Training example (LoRA)

python air_llm/examples/train_qwen38_flash_next_lora.py \
  --data my_data.jsonl \
  --seq-len 512 \
  --epochs 1 \
  --save-adapter qwen38-flash-next-lora.pt

The script streams each decoder layer, keeps adapters in GPU memory, and writes a LoRA checkpoint.

Performance highlights (from the README)

Model	Parameters	VRAM needed (approx.)
Qwen3‑32B	32 B	~4 GB
Qwen3.8‑Flash‑Next (125 B MoE)	125 B	~5.95 GB
DeepSeek‑V3	671 B	~12 GB
Llama 3.1‑405B	405 B	~8 GB
Kimi K3 (2.8 T)	2.8 T	~3.7 GB

Who might use it

• Researchers or hobbyists with a single mid‑range GPU who want to experiment with state‑of‑the‑art LLMs.
• Developers needing to run inference for large models in low‑cost environments (e.g., edge servers, personal workstations).
• Anyone looking for a drop‑in replacement for transformers.AutoModel that dramatically lowers VRAM requirements.

Where to get it

• Install via PyPI: pip install airllm.
• Source code, notebooks, and model‑specific scripts are on GitHub (https://github.com/lyogavin/airllm).
• Community chat on Discord and a Patreon for support.

---

All information above is taken directly from the repository’s README; no external claims have been added.