AirLLM enables 70B LLM inference on a single 4GB GPU

Overview

AirLLM makes it possible to run very large language models on consumer‑grade GPUs by keeping only one model layer on the device at a time.

How AirLLM Works

The library splits a model into individual layers and streams each layer from disk to the GPU only when it is needed for computation. This means the VRAM requirement is determined by the size of the largest layer, not the total number of parameters.

Supported Models and VRAM Requirements

AirLLM works with a wide range of open LLMs. The table below shows the approximate VRAM needed for various models when using the default (no‑compression) mode:

  • Qwen3, Mistral, Phi (≈8 B) – ~1–2 GB
  • Qwen3‑30B, Mixtral (MoE) – ~1–3 GB
  • Qwen3‑235B (MoE) – ~3 GB
  • Llama 3.x 70B (full precision) – ~4 GB
  • Llama 3.1 405B – ~8 GB
  • DeepSeek‑V3 (671 B) – ~12 GB
  • Kimi K3 (2.8 T) – ~3.72 GB VRAM measured on an RTX 6000 Ada (requires additional dependencies)

Performance Characteristics

Running a model in the default mode incurs high latency because each token triggers a layer load from disk. The library offers an optional block‑wise compression that can speed up inference by up to 3× with negligible accuracy loss. A user‑reported measurement for Kimi K3 on an RTX 6000 Ada shows approximately 292 seconds per token.

Setup and Usage

  1. Install the package: pip install airllm.
  2. Initialize a model with AutoModel.from_pretrained(<repo_id>).
  3. (Optional) Enable compression: add compression='4bit' or compression='8bit' and ensure bitsandbytes is installed.
  4. (Optional) Provide a Hugging Face token for gated models via the hf_token argument.
  5. (Optional) Set layer_shards_saving_path to control where the split layers are stored.
  6. (Optional) Turn on profiling_mode=True to see timing breakdowns.
  7. Run generation as shown in the quickstart example.

Note: The first run decomposes the model and saves layer shards, so sufficient free disk space in the Hugging Face cache directory is required.

Platform Support

  • Linux and Windows: standard PyTorch installation.
  • macOS: only Apple Silicon is supported; you must install mlx and torch as described in the macOS section.

Limitations and Trade‑offs

  • Latency: layer‑by‑layer streaming makes interactive chat slow; the technique is better suited for batch or overnight jobs.
  • Disk usage: storing all layer shards can consume significant space; you can enable delete_original=True to keep only the transformed version.
  • No quantization by default: the base approach avoids quantization to preserve accuracy; compression is an optional add‑on.
  • Certain models (e.g., Kimi K3) have extra requirements such as compressed-tensors, flash-attn, a CUDA 12 build of PyTorch, and transformers 4.56.x.

Community Perspective

Commenters on Hacker News highlighted both the excitement and the practical concerns:

  • "Kimi K3 on RTX 6000 Ada (48GB) takes 292 s/token" – @imenani
  • "Seeing a lot of these ‘run 1TB models with 1GB RAM’ projects recently. Most seem vibe coded and probably won’t be maintained." – @roger_
  • "I love it how the rampocalypse is pushing people to squash all the performance they can. I hope that this also leads to rethinking model architecture." – @seu
  • "I’m still slightly confused on what this adds… Does this basically load layers in and out on demand? So I still have to download the full model to disk…" – @cpfohl
  • "You can run any frontier model on your PC if you just wait long enough…" – @xg15
  • "The layer‑by‑layer streaming is clever. Curious how the throughput compares to running a quantized model on the same GPU — seems like quantization might still win on speed?" – @junsu22
  • "‘Can run’ and ‘is useful interactively’ are different benchmarks. At this latency, I can still imagine batch or overnight jobs being interesting; for chat, time to first useful answer matters much more…" – @Alisaqqt

These remarks underscore that while AirLLM achieves impressive memory efficiency, the resulting latency limits interactive use and invites comparison with quantized inference engines such as llama.cpp or unsloth.

Sources