Headmaster218/MoE4All

What it solves

MoE4All allows users to run large Mixture-of-Experts (MoE) AI models that exceed the capacity of their local GPU VRAM. It enables high-performance inference on consumer hardware (AMD, NVIDIA, and Intel) by intelligently managing model weights across SSD, system RAM, and GPU VRAM.

How it works

It uses a tiered storage architecture that treats SSD, RAM, and VRAM as a unified memory pool. Because MoE models only activate a small subset of their total parameters per token, the engine keeps frequently used 'expert' weights in VRAM while offloading the rest to RAM or SSD. It utilizes Vulkan for cross-platform GPU acceleration and provides an OpenAI-compatible API for integration with existing chat clients.

Who it’s for

This project is for users with consumer-grade GPUs who want to run large, high-parameter LLMs locally without needing enterprise-grade hardware. It is particularly useful for those who want to experiment with long-context models or perform concurrent inference tasks on limited VRAM.

Highlights

• Cross-Platform Support: Runs on Windows, Linux, and Apple Silicon with minimal setup.
• Vulkan-First Inference: Leverages Vulkan for broad compatibility across AMD, NVIDIA, and Intel GPUs.
• Efficient Memory Management: Implements a three-tier caching system (SSD/RAM/VRAM) to handle models larger than physical VRAM.
• API Compatibility: Provides an OpenAI-compatible API for embedding and chat services.
• Concurrent Processing: Supports multiple K/V slots for handling simultaneous requests with varying context lengths.