sqliteai/warp

What is WARP?

WARP (Weight‑Aware Runtime and Paging) is a tiny, self‑contained inference engine written in C. It lets you run very large mixture‑of‑experts (MoE) language models – like the 2.78‑trillion‑parameter Kimi K3 – on a laptop or desktop without needing a GPU or any external libraries.

How it works

• Hybrid memory layout – The shared part of a model (the “trunk”) stays in RAM, while the massive expert weights live on fast storage (SSD). When a token is processed the engine predicts which experts will be needed, streams those expert blocks from disk, and caches them in a bounded RAM‑resident expert cache.
• Look‑ahead routing – A lightweight router guesses the next‑layer experts early so the disk reads can overlap with computation, reducing idle time.
• Quantisation – Shared weights are kept at 4‑ or 8‑bit precision; experts use 3‑bit residual vector quantisation, shrinking the on‑disk footprint.
• Efficient KV cache – Linear attention and a compressed key‑value cache keep context memory tiny (≈0.2 GB for a 4 K context).

What you can run

Model	Size (container)	Minimum RAM	Speed (tokens / s)
Kimi K3 2.78 T	982 GB	29 GB	0.45 – 0.62 (64‑token run)
DeepSeek‑V4.1‑Flash 552 B	299 GB	4.9 GB	3.7
GLM‑5.3‑Flash 313 B	112 GB	5.1 GB	3.3 – 3.9
Kimi‑Linear 48 B	19 GB	1.3 GB	14 – 17

The engine can also handle multimodal inputs – you can attach images with --image and the model’s vision tower will process them.

Getting started

1. Build – make compiles the waste CLI and a static library (libwaste.a). No BLAS, CUDA, Python, or other runtime dependencies are required.
2. Run – Example command:

   ./waste run ~/models/k3.waste "What is the capital of Italy?"
   

   The CLI reports memory usage, cache hit/miss rates, and throughput.
3. Convert models – A Python conversion script (convert.py) turns public checkpoints (safetensors, fp8, etc.) into the WARP container format. Conversion needs extra temporary storage (up to ~1.5 TB for Kimi K3) but the resulting container can be used on any machine that meets the RAM/storage requirements.

Why it matters

• Local inference for frontier models – Most 2‑trillion‑parameter models require dozens of gigabytes of GPU memory. WARP shows that, by streaming experts from SSD, you can run such models on consumer‑grade CPUs.
• Open and embeddable – The engine is a single C library with a permissive build, making it easy to embed in other applications or services.
• Human‑in‑the‑loop development – The codebase is generated by LLMs under human direction, illustrating a novel workflow for rapid prototyping of new inference algorithms.

Who should look at this?

• Researchers exploring CPU‑only inference for massive MoE models.
• Developers who need an offline, privacy‑preserving LLM runtime (no cloud, no GPU drivers).
• Anyone curious about weight‑aware paging techniques and how to squeeze the most out of modern SSDs for AI workloads.

---

TL;DR: WARP is a C‑only, zero‑dependency engine that streams MoE expert weights from disk, letting you run trillion‑parameter language models on a laptop with modest RAM. It offers a CLI, a static library, and support for text‑plus‑image generation.