WASTE Inference Engine: Running Kimi K3 2.78T on Consumer Hardware
WASTE (Weight-Aware Streaming Tensor Engine) enables the execution of frontier-scale models, specifically the 2.78-trillion-parameter Kimi K3, on consumer hardware by streaming activated weights directly from NVMe storage. It allows Kimi K3 to run on a machine with as little as 29.05 GB of RAM, achieving a decode speed of approximately 0.5 tokens per second on a 64 GB MacBook Pro.
Architecture: NVMe Streaming and Weight Management
WASTE overcomes the memory limitations of consumer hardware by treating the disk as the primary storage for the model's Mixture of Experts (MoE) weights, keeping only the model trunk resident in RAM.
Weight-Aware Streaming
Because a Mixture of Experts model only activates a small fraction of its parameters per token (approximately 4% for K3), WASTE streams only the required experts from disk for each token. To minimize I/O overhead, the model is converted into a .waste container where each expert record is 4 KiB-aligned. This layout ensures that routing to an expert requires exactly one pread operation, bypassing the OS page cache using F_NOCACHE (macOS), O_DIRECT (Linux), or FILE_FLAG_NO_BUFFERING (Windows) to prevent the kernel from attempting to cache a terabyte-scale model.
Residual Vector Quantization (RVQ)
To reduce the disk footprint and I/O bandwidth requirements, experts are stored using three-stage residual vector quantization at 3.00 bits per weight. The engine performs arithmetic directly on these quantized weights without materializing the full matrix, reducing each expert row operation to three table reads and two additions.
Memory Budgeting and the "Cache Floor"
WASTE utilizes remaining RAM as a bounded expert cache. The engine identifies a "cache floor"—the minimum RAM required to hold one token's working set (17.0 GB for K3).
- Below the floor: Cache hit rates are 0%, as experts are evicted before the next token can request them.
- Above the floor: Hit rates increase sharply, improving throughput.
- The Paging Ceiling: If the RAM budget is set too high (e.g., 58 GB on a 64 GB machine), the OS begins paging the expert cache to disk, causing a massive performance drop (from 0.32 tok/s down to 0.04 tok/s).
To avoid this, WASTE automatically calculates a budget that fits under seven-eighths of physical RAM, stepping down by whole working sets to maximize efficiency without triggering OS paging.
Performance Benchmarks
Benchmarks were conducted on a MacBook Pro M5 Pro with 64 GB of RAM and an internal NVMe SSD.
Kimi K3 (2.78T Parameters)
| Metric | Value |
|---|---|
| Minimum RAM | 29.05 GB (at 4K context) |
| Container Size | 982 GiB |
| Decode Speed | 0.49–0.54 tok/s |
| Resident Trunk | 27.28 GB |
| Read per Token | 17.0 GB |
| Vision Tower | 15.7 s for 1024-patch image |
Kimi-Linear (48B Parameters)
| Metric | Value |
|---|---|
| Minimum RAM | 1.87 GB |
| Container Size | 19 GB |
| Decode Speed | 10.7 tok/s |
Technical Implementation Details
Linear Attention and KV Cache Optimization
K3 uses a hybrid attention mechanism (Kimi Delta Attention and gated multi-head latent attention). WASTE absorbs the kv_b_proj into the query and output, reducing the KV cache size by 53×. At a 4K context, the cache requirement drops from 11.25 GB to 0.21 GB, enabling significantly longer context windows on limited hardware.
Engine Specifications
- Language: Written in C11 with no third-party runtime dependencies (only
libcandpthreads). - Embeddability: The engine is provided as a library (
libwaste.a) with 26 public functions, making it fully embeddable in other applications. - Multimodal Support: Includes a 401M ViT (27 layers) for image processing. Images are converted into embeddings that are processed as text tokens through the MoE layers.
- Platform Support: Supports macOS (ARM64), Linux (ARM64/x86_64), and Windows (x86_64 via MinGW-w64).
Community Insights and Counterpoints
While the technical achievement of running a 2.78T model on a laptop is significant, community discussion highlights several practical trade-offs:
"At 0.5t/s and reading many gigabytes from the SSD every second... I wonder if this wouldn't be extremely practical if targeting a 500gib or 250gib model..."
Critics point out that 0.5 tokens per second is extremely slow for practical use, particularly for "thinking" models like K3 which are often verbose. Others noted the high energy cost per token compared to GPU clusters and the massive storage requirement (1 TB) as a barrier to entry for many consumer users.