jesdga95/basalt

What it solves

Basalt allows you to run the massive Qwen3.8-Flash-Next mixture-of-experts model on a single workstation. It solves the problem of memory constraints by intelligently offloading parts of the model that don't fit in your GPU's video memory (VRAM) to your system RAM or a secondary GPU.

How it works

Basalt acts as a local inference engine that serves the model via an API compatible with OpenAI and Anthropic standards. It uses a custom file format that packages weights, tokenizers, and expert profiles into a single file. When running, it dynamically manages "experts"—the specialized components of the model—keeping the most active ones in VRAM while utilizing system RAM for the rest. It is designed specifically for NVIDIA Blackwell GPUs.

Who it’s for

This tool is for developers and power users with high-end hardware (specifically NVIDIA Blackwell GPUs) who need to run large, complex AI models locally without needing a massive server cluster.

Highlights

• Runs large mixture-of-experts models on a single workstation.
• Provides an OpenAI- and Anthropic-compatible API for easy integration.
• Supports offloading model components to system RAM or a second GPU.
• Includes a built-in web interface for monitoring and interaction.
• Offers specialized model packs optimized for different performance and quality needs.