Helldez/BigMoeOnEdge

Run MoE models bigger than your RAM. Frontier-size MoE on a 12 GB phone, CPU only, lossless, on stock llama.cpp

What it solves

BigMoeOnEdge allows users to run large Mixture-of-Experts (MoE) models on devices with limited RAM, such as smartphones or PCs using only the CPU. It solves the problem where models larger than the available system memory would typically refuse to load or cause the OS to crash/slow down due to excessive swapping (mmap fault storms).

How it works

Instead of loading the entire model into RAM, the engine treats flash storage as part of the memory hierarchy. It keeps only the essential, always-needed parts of the model resident in memory and streams only the specific "experts" required for each generated token directly from flash storage at the moment they are needed.

It is built on top of the llama.cpp public API, meaning it supports all quantization formats and tokenizers that llama.cpp does. It can handle multi-shard GGUF files natively without requiring a merge step.

Who it’s for

Developers and AI enthusiasts who want to run massive MoE models (e.g., 100B+ parameters) on consumer hardware with tight memory constraints, specifically on Android phones or CPU-only PCs.

Highlights

  • Extreme Memory Efficiency: Enables running models like DeepSeek V4 Flash (284B parameters, ~91 GB) on a phone with only 12 GB of RAM.
  • Lossless Streaming: By default, the output is byte-identical to running the model fully resident in RAM.
  • Performance Tuning: Includes a variety of "knobs" to balance speed and quality, such as expert caching, parallel I/O lanes, and lossy optimizations like dropping "cold" experts or reducing the number of active experts.
  • Android App: Provides a ready-to-use chat application with a model downloader and live telemetry to monitor tokens per second and cache hit rates.
  • Broad Architecture Support: Supports a wide range of MoE architectures including Qwen, Gemma, DeepSeek, and Liquid AI LFM2.

Related

  • Project
    EfficientMoE/MoE-Infinity

    MoE-Infinity is a library for cost-effective MoE inference that enables running large Mixture-of-Experts models on memory-constrained GPUs by offloading expert weights to host memory and SSD.

  • Dispatch
    Mixture of Experts (MoE) Explained

    Mixture of Experts (MoE) allows transformer models to scale parameters while maintaining efficient pretraining and faster inference by activating only a subset of neural network experts per token.

  • Project
    FlashML-org/FreeToken

    An edge-native MoE serving engine that enables running 290B+ parameter frontier models on consumer gaming PCs and laptops using bandwidth-adaptive execution and elastic memory management.

  • Project
    kvcache-ai/ktransformers

    A framework for efficient LLM inference and fine-tuning using CPU-GPU heterogeneous computing, specifically optimized for ultra-large MoE models on limited hardware.

  • Project
    microsoft/Tutel

    An optimized Mixture-of-Experts (MoE) implementation that accelerates training and inference for massive LLMs on NVIDIA and AMD GPUs using advanced parallelism and quantization.