xiaol/wkvm

What it solves

WKVM is a hypervisor for model state designed for linear and hybrid-linear models (such as RWKV-7, GDN, and Mamba2). It replaces the traditional paged KV block chain—which can be memory-intensive for long contexts—with fixed-size per-request state slots. This allows for constant, tiny per-request memory footprints, simplifying scheduling and enabling advanced state management.

How it works

Instead of managing complex KV caches, WKVM treats model states as virtual machines. It allocates a fixed-size state slot for each request, allowing the engine to perform operations like creating, snapshotting, forking, hibernating, and resuming states. For hybrid models, full-attention layers are handled in a simple paged guest pool. It features a "Durable State API" that allows state handles to be named, versioned, and exported across different storage tiers (GPU, host memory, and NVMe).

Who it’s for

It is intended for developers and researchers working with recurrent or linear-attention models who need high-throughput inference, efficient state management, and the ability to handle thousands of concurrent sessions with minimal memory overhead.

Highlights

• State-Native Inference: Treats model states as first-class objects that can be migrated, forked, or hibernated.
• Durable State API: Supports named, versioned, and mutable state handles stored across GPU, pinned host memory, and NVMe.
• High Throughput: Demonstrates significant throughput gains over vLLM and SGLang in specific linear/hybrid model workloads.
• Efficient Scheduling: Uses a no-phases continuous-batching scheduler with exact admission based on free slot counting.