NotPunchnox/rkllama

Ollama alternative for Rockchip NPU: An efficient solution for running AI and Deep learning models on Rockchip devices with optimized NPU support ( rkllm )

What it solves

RKLLama provides a way to run and interact with Large Language Models (LLMs) directly on the NPU (Neural Processing Unit) of Rockchip RK3588 and RK3576 hardware. This allows users of single-board computers like the Orange Pi 5 series or Radxa Rock 4d to perform AI inference locally with hardware acceleration, rather than relying solely on the CPU.

How it works

The project acts as a server that manages model loading, inference, and API requests. It uses the rkllm and rknn runtime libraries to execute models on the NPU. It includes a REST API that is compatible with Ollama and OpenAI standards, allowing it to integrate with existing AI tools and web interfaces. It handles model management dynamically, loading models into memory as needed and unloading them after inactivity to optimize resource usage.

Who it’s for

This project is for developers and hobbyists using Rockchip-based hardware (RK3588/RK3576) who want to run LLMs, vision models, or audio processing tasks locally. It is ideal for those who want to use standard AI API tools (like OpenWebUI) on low-power, edge-computing devices.

Highlights

  • Runs LLMs, vision models, and audio models (TTS/STT) directly on the NPU.
  • Compatible with common Ollama and OpenAI API endpoints.
  • Supports tool/function calling for advanced AI interactions.
  • Features dynamic model loading and unloading to manage memory efficiently.
  • Includes automatic prompt caching to speed up repeated chat sessions.
  • Supports multimodal inputs (images) and image generation via Stable Diffusion.
  • Offers easy model installation directly from Hugging Face.

Related

  • Project
    mlc-ai/web-llm

    WebLLM is a high-performance inference engine that enables large language models to run directly in web browsers using WebGPU, removing the need for server-side processing.

  • Project
    alibaba/rtp-llm

    A high-performance LLM inference acceleration engine developed by Alibaba that optimizes throughput and latency for production-scale deployments.

  • Project
    vllm-project/vllm

    vLLM is an open‑source library for high‑throughput, low‑latency serving of large language models. It introduces PagedAttention for efficient KV‑cache handling, supports many quantisation formats, integrates fast attention kernels, and offers an OpenAI‑compatible API that runs on GPUs, CPUs and other accelerators.

  • Project
    bentoml/OpenLLM

    OpenLLM is a Python tool that lets you run open‑source LLMs (e.g., Llama 3, Qwen 2.5, Mistral) locally and expose them through OpenAI‑compatible REST APIs. With a single command you can start a GPU‑accelerated server, use a built‑in web chat UI, and deploy the service via Docker or Kubernetes. It supports a catalog of models, custom model repositories, and integrates with the high‑performance vLLM backend.

  • Project
    noonghunna/club-3090

    A collection of optimized Docker recipes and benchmarks for running LLMs locally on NVIDIA RTX 3090 and similar consumer GPUs, providing an OpenAI-compatible API for easy integration.