FlashML-org/FreeToken

FreeToken – Edge‑Native Mixture‑of‑Experts (MoE) Serving Engine

What it is – FreeToken is an open‑source runtime for running very large Mixture‑of‑Experts language models (e.g., 290 B‑parameter models) on consumer‑grade hardware such as gaming PCs and laptops. It treats CPUs, GPUs, host memory and interconnects as a single elastic inference platform, letting you serve frontier‑scale open‑weight models without a data‑center.

Key capabilities

• Bandwidth‑adaptive CPU‑GPU co‑execution – a custom $q^\star$ scheduling policy moves work between CPU and GPU based on current bandwidth, keeping inference fast.
• Double‑buffered pre‑fill streaming – layers are streamed in a pipelined fashion, reducing latency for long prompts.
• Global LRU expert cache & semantic‑aware checkpoints – frequently used experts and KV‑cache states are kept in a least‑recently‑used cache; semantic anchors let you edit context (e.g., tool calls) without recomputing the whole prompt.
• Elastic VRAM management – the engine can re‑allocate GPU memory on‑the‑fly between expert weights and KV memory, avoiding restarts.
• Broad model & format support – works with open‑weight MoE models such as DeepSeek‑V4‑Flash, Qwen3.6‑35B‑A3B, GLM‑5.2, and accepts quantizations like MXFP4, NVFP4, FP8, BF16.
• API compatibility – provides Anthropic/OpenAI‑style endpoints, so existing agents (Codex, Claude Code, etc.) can call the engine unchanged.
• Hardware coverage – native support for NVIDIA RTX 30/40/50 series GPUs; also runs on CPUs when GPUs are unavailable.

How to start

1. Desktop app – download a pre‑built Windows or Linux package from https://flashml.ai. The GUI lets you load models, chat, and tweak runtime settings.
2. CLI – install with the modern Python package manager uv:

   uv pip install "freetoken[accel]"
   

   or build from source:

   git clone https://github.com/FlashML-org/FreeToken.git && cd FreeToken
   uv venv && source .venv/bin/activate
   uv pip install -e ".[accel]"
   

   Detailed docs are in the docs/ folder (install, quick‑start, supported models, CLI reference, checkpoint hot‑fixes).

Why it matters – Traditional MoE serving requires multi‑GPU clusters with high‑speed interconnects. FreeToken brings that capability to the edge, enabling interactive AI applications (code assistants, tool‑calling agents, etc.) on a single consumer machine while keeping memory usage and latency low.

Citation – If you use the engine in research, cite the accompanying arXiv paper (arXiv:2608.16157).

Acknowledgments – The project builds on ideas from mini‑sglang, vLLM, FlashInfer, LightLLM, and llama.cpp, and is released under the Apache‑2.0 license.