AirLLM enables 70B LLM inference on a single 4GB GPU

Overview

AirLLM reduces inference memory usage so that a 70B parameter model can run on a single 4 GB GPU without quantization, distillation, or pruning. The library works by loading only one model layer onto the GPU at a time, making VRAM consumption depend on layer size rather than total model size.

How AirLLM reduces VRAM usage

AirLLM keeps exactly one layer of the model on the GPU during inference. When a new layer is needed, the previous layer is evicted and the next layer is loaded from disk. This layer‑by‑layer streaming means the required VRAM is roughly the size of the largest layer, which for many LLMs is a few gigabytes regardless of the total parameter count.

Supported models and VRAM requirements

AirLLM supports virtually every popular open LLM family, including Llama, Qwen, DeepSeek, Mistral, Mixtral, Phi, Gemma, ChatGLM, Baichuan, InternLM, and Yi. The README provides typical VRAM usage for various models when run with AirLLM:

  • Qwen3, Mistral, Phi (~8B) → ~1–2 GB
  • Qwen3‑30B / Mixtral (MoE) → ~1–3 GB
  • Qwen3‑235B (MoE) → ~3 GB
  • Llama 3.x 70B (full precision) → ~4 GB
  • Llama 3.1 405B → ~8 GB
  • DeepSeek‑V3 (671B) → ~12 GB These figures are measured end‑to‑end on a single GPU; the same one‑line AutoModel.from_pretrained call works for all listed models.

Model compression for speed‑up

Starting with version 2.0, AirLLM offers optional block‑wise quantization (4‑bit or 8‑bit) that can speed up inference by up to 3× with negligible accuracy loss. To enable compression, install bitsandbytes and pass compression='4bit' or compression='8bit' when initializing the model:

from airllm import AutoModel
model = AutoModel.from_pretrained("garage-bAInd/Platypus2-70B-instruct", compression='4bit')

The compression only quantizes weights, which simplifies accuracy preservation compared to full weight‑and‑activation quantization.

Quick start usage

  1. Install the package: pip install airllm.
  2. Load any supported model with a single line:
from airllm import AutoModel
model = AutoModel.from_pretrained("Qwen/Qwen3-32B\))  # replace with desired HF repo ID
  1. Tokenize input and generate output as with a regular 🤗 Transformers model. The first run will decompose the model and save layer shards to the HuggingFace cache, so sufficient disk space is required.

Performance and latency (from comments)

A comment on the Hacker News thread notes that running Kimi K3 (2.8T) on an RTX 6000 Ada (48 GB) takes about 292 seconds per token, illustrating the trade‑off between low VRAM usage and high latency.

“For anyone wondering “how slow is this?” IIUC, Kimi K3 on RTX 6000 Ada (48GB) takes 292 s/token”

Comparison with other approaches (from comments)

Another comment questions whether AirLLM’s layer‑streaming approach offers advantages over existing tools such as llama.cpp with appropriate flags (-cmoe/-mmap) that also manage VRAM versus RAM versus SSD usage.

“What's the benefit of these projects over something like taking an unsloth quant and running llama.cpp with appropriate flags (-cmoe/-mmap) to manage VRAM vs RAM vs SSD?”

Limitations and FAQ

The README FAQ highlights several practical points:

  • MetadataIncompletebuffer error: usually caused by insufficient disk space during model splitting; clearing the HuggingFace cache or expanding disk space resolves it.
  • Wrong model class: loading QWen or ChatGLM models with a Llama‑specific class results in a ValueError; using AutoModel avoids this.
  • Gated models: provide a HuggingFace token via the hf_token argument.
  • Padding token missing: disable padding in the tokenizer call or set a padding token.

Conclusion

AirLLM makes it possible to run very large language models on modest GPUs by streaming one layer at a time, with optional compression for speed‑up. While this enables access to models that would otherwise exceed GPU memory, the resulting latency can be high, as shown by the reported ~292 s/token for the largest models. Users must weigh VRAM savings against inference speed and consider alternative quantization‑based tools depending on their workload.

Sources