FlashML-org/FreeToken

What it solves

FreeToken is designed to run massive Mixture-of-Experts (MoE) models (up to 290B+ parameters) on consumer-grade hardware, such as gaming PCs and laptops, at interactive speeds. It overcomes the memory and bandwidth limitations of edge devices by treating GPUs, CPUs, and host memory as a unified, elastic platform for inference.

How it works

The engine uses several optimization techniques to maximize hardware efficiency:

• Bandwidth-Adaptive Execution: A $q^\star$ policy for CPU-GPU co-execution to optimize data movement.
• Expert Caching: A global LRU (Least Recently Used) expert cache and the FTW fast weight format to speed up model weights access.
• Semantic-Aware Caching: Uses semantic anchor checkpoints for KV caches and recurrent states, preventing redundant recomputation during agentic context edits (like tool calls).
• Elastic Memory: Dynamically re-allocates VRAM between expert caches and KV memory at runtime without requiring a restart.
• Quantization Support: Supports multiple formats including MXFP4, NVFP4, FP8, and BF16 to reduce the memory footprint.

Who it’s for

Developers and AI enthusiasts who want to run frontier-scale open-weight MoE models locally on NVIDIA RTX 30, 40, and 50 series GPUs without relying on cloud infrastructure.

Highlights

• Edge-Native Runtime: Optimized for consumer hardware with double-buffered prefill streaming and graph-compatible execution.
• Broad Model Support: Compatible with models like DeepSeek-V4-Flash, Qwen3.6-35B-A3B, and GLM-5.2.
• API Compatibility: Provides Anthropic and OpenAI-compatible APIs for easy integration with coding agents like Claude Code and DeepSeek Harness.
• Cross-Platform: Available as a desktop app for Windows and Linux or via a CLI.