brontoguana/krasis

Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware

What it solves

Krasis allows users to run massive Mixture-of-Experts (MoE) models—some with hundreds of billions of parameters—on consumer-grade NVIDIA GPUs. It solves the problem of these models being too large to fit into standard GPU VRAM by managing how model weights are stored and moved between system RAM and VRAM.

How it works

Krasis uses a high-performance runtime written in Rust and CUDA to orchestrate model execution. It employs several key technologies to reduce memory footprints and increase speed:

  • HCS Expert Residency Management: Dynamically manages "hot" and "cold" experts, moving them between VRAM and CPU RAM so models larger than the available VRAM can still function.
  • Quantization: Builds cached INT4/INT8 expert formats and HQQ attention caches to compress model weights.
  • Compact KV Caching: Offers specialized modes (like k6v6 and k4v4) to reduce the memory used by the Key-Value cache.
  • GPU-Accelerated Processing: Uses full GPU prefill for fast prompt processing and GPU-executed decode.

Who it’s for

It is designed for developers and AI enthusiasts with NVIDIA GPUs (Ampere architecture and newer) who want to run state-of-the-art, large-scale MoE models locally without needing enterprise-grade hardware.

Highlights

  • Broad Model Support: Validated for families including DeepSeek-V4, Qwen3, Gemma 4, and Nemotron MoE.
  • OpenAI Compatibility: Provides an OpenAI-compatible API and translates native model tool-call syntax into structured tool_calls.
  • Krasis Manager: A Rust-based dashboard for monitoring GPU VRAM use and managing model configurations via a JSON API.
  • Cross-Platform: Native installers for Windows and support for Linux/WSL2.
  • Performance Tooling: Includes built-in repeatable benchmarks and a dedicated chat client.

Related

  • Project
    kacper-daftcode/vLLM-Moet

    vLLM‑Moet is a patched vLLM fork that adds 2‑bit expert quantisation, FP4 recovery, tiered GPU/host/NVMe expert storage, and speculative decoding to run massive MoE models (e.g., 753 B GLM‑5.2) on consumer‑grade RTX PRO 6000 or RTX 5090 GPUs. It ships Docker images, custom SM 120 kernels, and a set of configurable knobs for memory‑speed‑quality trade‑offs.

  • Project
    microsoft/Tutel

    Tutel is Microsoft’s open‑source, high‑performance MoE library for large language models. It provides dynamic parallelism, low‑precision inference (NVFP4, MXFP4, FP8, BF16), support for up to 1 M‑token contexts, vision extensions, and Docker images that serve models like DeepSeek‑V3.2, Kimi‑K3, GLM‑5.x, Qwen‑3, and GPT‑OSS on both NVIDIA and AMD GPUs. Install via pip from the Git repo or build from source, then run the supplied Docker containers or use the Python API for custom MoE routing.

  • Project
    guqiong96/Lvllm

    An extension of vLLM that enables hybrid GPU and NUMA-aware CPU inference for MoE models, allowing them to span across VRAM and system memory.

  • Project
    EfficientMoE/MoE-Infinity

    A cost-effective inference library that enables running large Mixture-of-Experts models on memory-constrained GPUs by offloading expert weights to host memory and SSD.

  • Project
    kvcache-ai/ktransformers

    A framework for efficient LLM inference and fine-tuning that uses CPU-GPU heterogeneous computing to run ultra-large MoE models on consumer hardware.