mw00/project-maya

What it solves

Project Maya allows users to run the massive GLM-5.3-Flash model (321 billion parameters) on consumer-grade hardware. Normally, models of this scale require enterprise servers with hundreds of gigabytes of GPU memory; Maya enables this by implementing a tiered memory system that distributes model experts across VRAM, RAM, and NVMe SSDs, moving them as needed during a conversation.

How it works

The engine uses a Mixture-of-Experts (MoE) architecture where only a fraction of the total parameters (about 18B) are active per token. Maya optimizes this by keeping the most frequently used experts in the GPU's VRAM, the next most used in system RAM, and the remainder on a fast NVMe SSD. It supports speculative decoding via an MTP block to draft tokens ahead and provides several quantized versions of the model (Maya-S, S24, M, M-Derisked, and L) to balance accuracy and hardware requirements.

Who it’s for

It is designed for AI enthusiasts and developers with NVIDIA GPUs (V100 or newer) or specific AMD GPUs (experimental) running Linux or Windows, who want to run a high-parameter model locally without needing a massive GPU cluster.

Highlights

• Tiered Memory Management: Dynamically swaps experts between VRAM, RAM, and SSD to fit huge models on limited hardware.
• Multi-GPU Support: Can scale across up to 16 GPUs, splitting layers based on available VRAM.
• OpenAI & Anthropic Compatible: Provides an API that works with existing tools and coding agents.
• Vision Capabilities: Includes an image encoder for multimodal inputs.
• Custom Quantizations: Offers multiple model sizes (from 96.5 GB to 156.3 GB) tailored to different memory pools.