prabanta-dev/janas

37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.

What it solves

Janas-LLM addresses the challenge of running massive, high-parameter mixture-of-experts (MoE) language models on consumer hardware with limited RAM. By streaming model weights from an NVMe SSD and caching frequently used experts in memory, it enables large models (such as 80B parameter variants) to run on standard laptops with 32GB of RAM.

How it works

Written in C with no external dependencies, the engine uses a custom architecture to manage model weights. It keeps the bulk of the model on disk and dynamically loads only the required experts into an active memory cache. It leverages hardware acceleration (Vulkan) where available and uses direct I/O (O_DIRECT) to maximize disk read speeds, allowing it to perform inference at speeds often exceeding established frameworks like llama.cpp on specific hardware configurations.

Who it’s for

This project is for developers and power users who want to run large-scale LLMs locally on consumer-grade hardware (Linux or Windows x86-64) and are interested in high-performance, low-dependency C implementations of AI inference engines.

Highlights

  • Runs 80B parameter models on 32GB RAM laptops by streaming experts from disk.
  • High-performance C implementation with no external library dependencies.
  • Includes built-in support for MCP (Model Context Protocol) to integrate with AI agents and editors.
  • Provides an OpenAI-compatible HTTP server and code-completion capabilities for IDEs like VS Code.
  • Includes native benchmarking tools to measure and optimize performance on specific hardware.

Related

  • Project
    eliasbiondo/antirion

    Antirion is a fast, single-binary AI gateway that unifies access to many model providers, with routing, cost control, safety, and monitoring, deployable via Docker, Kubernetes, or binaries.

  • Project
    ollama4j/ollama4j

    Ollama4j is a Java library that wraps the Ollama local LLM server, providing text generation, chat, tool calling, embeddings, image inputs, model management, and Prometheus metrics. It is distributed via Maven Central/Gradle and integrates with any Java project.

  • Project
    xybrid-ai/xybrid

    Xybrid provides native, offline SDKs (Flutter, Swift, Kotlin, React Native, Unity, Rust, Python) to run LLMs, ASR, TTS and vision models on‑device, with multi‑model pipelines and tool‑calling support.

  • Project
    ultralytics/yolo-flutter-app

    YOLO Flutter is Ultralytics' official Flutter plugin that enables on‑device YOLO inference (detection, segmentation, depth, classification, pose, OBB) for both iOS and Android. It provides a single Dart API—`YOLO` for single‑image inference and `YOLOView` for live‑camera inference—handling automatic download and caching of official LiteRT/Core ML models, custom model loading, metadata‑driven task inference, and optional Qualcomm NPU support.