Indras-Mirror/Xolotl

What it solves

Xolotl allows users to run massive, frontier-level AI models—such as the 552-billion-parameter DeepSeek-V4.1—on a single consumer gaming PC. It solves the hardware barrier by using a tiered memory system that intelligently manages model data across VRAM, system RAM, and NVMe storage, rather than requiring expensive server-grade hardware.

How it works

The project uses a specialized engine that treats a model's "experts" (the specific parts of the model used for any given task) as a cache. It keeps the most frequently used experts in the GPU's VRAM, while storing the rest in system RAM or on a fast NVMe drive. As the model processes text, the engine predicts which experts are needed next and fetches them just in time. It also includes specific optimizations for different model architectures, such as speculative decoding and custom quantization, to maintain high performance on consumer hardware.

Who it’s for

Xolotl is designed for developers and power users with high-end NVIDIA GPUs (such as an RTX 4090) who want to run large, uncensored, or frontier-class language models locally for coding agents, tool-calling, or private AI tasks without relying on external APIs or server farms.

Highlights

• Runs massive Mixture-of-Experts (MoE) models like DeepSeek-V4.1 on a single 24GB VRAM card.
• Provides an OpenAI/Anthropic-compatible server interface for easy integration with existing AI tools.
• Supports long context windows (up to 512K tokens) with efficient memory management.
• Includes advanced features like conversation caching, batch processing, and multi-GPU layer splitting.
• Offers native support for both Linux and Windows environments.