Janas LLM: Running Large Mixture-of-Experts Models via SSD Streaming

Overview

Janas-LLM is an experimental inference engine designed to run large mixture-of-experts (MoE) language models on consumer hardware that lacks sufficient RAM to hold the entire model. By streaming experts from an NVMe SSD and caching frequently accessed weights in memory, Janas-LLM achieves significantly higher token generation speeds compared to traditional CPU-based inference methods on similar hardware.

Core Architecture and Performance

Janas-LLM is written in pure C with no external dependencies. Its primary innovation lies in its memory management strategy: rather than requiring the entire model to reside in RAM, it treats the SSD as an extension of the memory hierarchy.

On a test machine equipped with an Intel Core Ultra 9 185H, 32 GB of RAM, and an NVMe SSD, Janas-LLM demonstrated substantial performance gains over existing solutions like llama.cpp. For instance, when running the 80B parameter Qwen3-Next-80B-A3B model (which exceeds the system's 32 GB RAM capacity), Janas-LLM reached 24 tokens per second (tok/s) for writing, compared to 2.8 tok/s for llama.cpp. Prompt processing speeds also showed marked improvement, with Janas-LLM achieving 62 tok/s versus 16 tok/s.

Experimental Nature and Hardware Considerations

Janas-LLM is currently in an early, experimental development stage. The project documentation explicitly warns that the engine is "hard on the machine," utilizing O_DIRECT for high-speed disk reads and filling available memory with an expert cache.

Key considerations for users include:

• System Stability: Because the engine aggressively manages memory, it can push the operating system into swap, potentially causing desktop unresponsiveness. Users have reported that GPU driver resets can occur if the engine attempts to load more data than the GPU can handle.
• SSD Wear: The reliance on high-speed disk streaming has led to community discussions regarding the potential for increased SSD wear due to constant, high-volume read operations.
• Hardware Variability: Performance is highly dependent on the specific machine configuration. The project includes a janas-bench tool designed to measure local hardware performance, which the author encourages users to share to help optimize the engine for a wider range of hardware.

Community Insights

Discussion surrounding the project reflects a broader trend in the open-source AI community toward optimizing local inference for constrained hardware. While some community members noted that similar techniques have been explored in other projects, others highlighted the significance of having multiple, competing implementations of SSD-streaming inference engines.

"There are many of these stream-experts-from-SSD projects now. What's needed is a guide to all of them and maybe for someone to take the best features of each of them and merge them into one uber project." — Hacker News User

Some users expressed concerns regarding the practical usability of such tools while performing other daily tasks, such as browsing or coding, noting that heavy disk and memory usage can impact overall system performance. Others pointed toward the future potential of specialized hardware, such as dedicated accelerators with integrated NVMe storage, which could theoretically provide the bandwidth necessary to feed model weights directly to compute units without the bottlenecks inherent in current consumer architectures.