Helldez/BigMoeOnEdge

Run MoE models bigger than your RAM. Frontier-size MoE on a 12 GB phone, lossless, on stock llama.cpp

What it solves

BigMoeOnEdge allows users to run massive Mixture-of-Experts (MoE) models on devices with very limited RAM, such as smartphones or PCs using only the CPU. It solves the problem where models larger than the available system memory typically crash or run extremely slowly due to OS paging/swapping. By treating flash storage as part of the memory hierarchy, it enables the execution of models that are several times larger than the device's RAM without losing output quality.

How it works

The engine is built on top of the llama.cpp public API and utilizes a streaming approach to weight management. Instead of loading the entire model into RAM, it keeps only the essential non-expert weights resident and reads only the specific "experts" required for each generated token directly from flash storage at the moment they are needed.

Key technical mechanisms include:

  • Expert Caching: Keeps frequently used experts in RAM to reduce storage reads.
  • Parallel I/O Lanes: Reads multiple expert slices simultaneously to saturate flash storage bandwidth.
  • Direct I/O: Bypasses the OS page cache to prevent redundant memory usage.
  • I/O and Compute Overlap: Fetches the next required weights while the current layer is still computing.
  • Row-Gathered Tables: Streams dense tables (like token embeddings) from flash in bounded windows rather than loading them fully.

Who it’s for

Developers and enthusiasts who want to run state-of-the-art large MoE models (like DeepSeek V4 Flash or Qwen series) on edge devices (Android phones, laptops) without requiring high-end GPUs or massive amounts of RAM.

Highlights

  • RAM-Independent Execution: Run models (e.g., 284B parameter models) on devices with as little as 12 GB of RAM.
  • lossless Streaming: Output is byte-identical to running the model fully resident in memory.
  • Broad Compatibility: Supports all quantization formats, tokenizers, and chat templates supported by llama.cpp.
  • NPU Acceleration: Optional prefill acceleration on Snapdragon Hexagon NPUs.
  • Decision Mode: A specialized --decide mode that answers multiple-choice questions via a single prefill, bypassing slow token-by-token decoding.
  • Multi-shard Support: Natively reads multi-shard GGUF files without needing a merge step.

Related

  • Project
    EfficientMoE/MoE-Infinity

    MoE‑Infinity is a library that enables inference of large Mixture‑of‑Experts language models on GPUs with limited VRAM by off‑loading expert weights to CPU/SSD and using an activation‑aware cache, plus a production‑grade OpenAI‑compatible server.

  • Dispatch
    Mixture of Experts (MoE) Explained

    Mixture of Experts (MoE) allows transformer models to scale parameters while maintaining efficient pretraining and faster inference by activating only a subset of neural network experts per token.

  • Project
    Edge0-AI/Edge0

    edge0 is an open‑source framework for streaming inference of large sparse‑MoE language models. It streams expert weights from SSD, uses a prerouter to pre‑fetch experts, and recovers quantisation loss with LoRA adapters, enabling 35B‑parameter models to run on a laptop with < 3 GiB RAM. The repo ships Python, macOS, iOS, Android and Windows runtimes, two model tiers (35B and 8B), benchmarks, and a roadmap for a unified cross‑platform API.

  • Project
    FlashML-org/FreeToken

    FreeToken is an open‑source runtime that lets you serve huge Mixture‑of‑Experts language models (up to 290 B parameters) on a single consumer PC. It dynamically balances work between CPU and GPU, streams model layers, caches expert weights, and provides an OpenAI‑compatible API, making datacenter‑class inference possible on a gaming rig.

  • Project
    guqiong96/Lvllm

    An integration of vLLM and lk_moe that enables large MoE models to run across CPU and GPU memory using NUMA-aware scheduling.