Edge0-AI/Edge0

edge0 – Streaming MoE Inference for Consumer Devices

What it is
edge0 is an open‑source framework that lets you run large sparse‑Mixture‑of‑Experts (MoE) language models on everyday hardware – laptops, phones and even Windows PCs. It does this by streaming expert weights from an SSD only when they’re needed, keeping the amount of RAM used roughly equal to the number of active experts rather than the full model size.

Key tricks

Technique	Why it matters
SSD expert offload	Expert parameters are stored on disk and mmapped on‑demand, so a 35‑billion‑parameter model can run with < 3 GiB of RAM.
Prerouter	A tiny trained head predicts which experts will be needed one step ahead, overlapping the disk load with the forward pass. The authors report up to +59 % higher decode throughput.
Recover‑LoRA	The base model is quantised to int4 (4‑bit). LoRA adapters, trained by distillation from a full‑precision teacher, recover most of the lost quality. Adapters stay separate files, allowing multiple fine‑tuned sets without touching the base checkpoint.

Supported platforms

Platform	Language / Stack	Status
Python (macOS Apple Silicon)	Python + MLX	Ready – install via pip
macOS app & CLI	Rust	Open‑sourced Sep 30 2026
iOS app	Swift + MLX‑Swift	Open‑sourced Sep 30 2026
Android app & engine	Kotlin + native engine	Open‑sourced Sep 30 2026
Windows app & engine	C++ + Vulkan	Open‑sourced Sep 30 2026

A unified inference framework that hides these platform differences is planned for Q4 2026.

Model tiers

Tier	Checkpoint	Size / Experts	Prerouter K
edge0‑35b	Hugging Face	4‑bit, 40 layers, 256 experts	4
edge0‑8b	Hugging Face	4‑bit, 24 layers, 128 experts	8

Each checkpoint bundles the base weights, LoRA adapters and prerouter heads, so a single download gives a ready‑to‑run model.

Performance
Benchmarks run on a Mac mini M4 Pro (24 GB RAM).

• edge0‑35b: 14.9‑17.7 tokens / s decode, 2.9 GiB peak active memory.
• edge0‑8b: 23.9‑25.3 tokens / s decode, 1.0 GiB peak active memory.

Quality loss compared to the original fp16 models is modest (average ≈ 3 points on MMLU‑Pro, with some benchmarks even surpassing the base).

Getting started (Python)

# clone and create a venv
cd python
python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev,fetch]'

# download a model (the script pulls both base and adapters)
.venv/bin/python scripts/fetch_models.py --tier edge0-35b --target-dir models

# set env var or pass the path directly
export EDGE0_35B_MODEL=$PWD/models/edge0-35b

# quick demo
edge0 demo edge0-35b

# serve an OpenAI‑compatible endpoint
edge0 serve edge0-35b   # then POST to http://127.0.0.1:8000/v1/chat/completions

The same engine can be used from Python code via AutoEngine.from_pretrained(...).

Roadmap (Q4 2026)

• A single unified API that automatically selects the correct native runtime (iOS, Android, Windows, macOS, Python).
• CUDA backend for Linux/desktop GPUs.
• Support for next‑generation architectures (hybrid linear attention, N‑gram tables) that fit naturally with SSD streaming.
• Latent‑thinking optimisations that move expert routing from per‑token to per‑block, further reducing latency.

License
Apache‑2.0 (see LICENSE).

---

All details above are taken directly from the repository’s README.