NotPunchnox/rkllama

What it solves

RKLLama provides a way to run Large Language Models (LLMs) and other AI models specifically optimized for Rockchip RK3588(S) and RK3576 platforms. Unlike general-purpose tools like Ollama or Llama.cpp, RKLLama enables these models to run directly on the NPU (Neural Processing Unit), significantly improving performance on this specific hardware.

How it works

It operates as a server-client architecture. The server handles the loading and execution of .rkllm and .rknn models using the rkllm-runtime and rknn-runtime libraries. It provides a REST API that is compatible with Ollama and partially compatible with OpenAI, allowing users to interact with the models via standard API calls or a provided Python client. The system supports dynamic model loading/unloading, prompt caching for fast context switching, and automatic CPU detection for the supported Rockchip platforms.

Who it’s for

Developers and hobbyists using Rockchip-based single-board computers (such as Orange Pi 5 series or Radxa Rock 4d) who want to run local AI models including text, vision, image generation, and speech tasks with NPU acceleration.

Highlights

• NPU Acceleration: Runs models on the NPU rather than just the CPU/GPU.
• Broad API Compatibility: Supports Ollama and partial OpenAI API endpoints for chat, completions, embeddings, and multimodal tasks.
• Multimodal Capabilities: Supports vision models (e.g., Qwen2VL) for image questioning, image generation (Stable Diffusion), and speech-to-text/text-to-speech (Whisper, Piper, MMS-TTS).
• Advanced LLM Features: Includes tool/function calling support for models like Qwen and Llama 3.2+.
• Efficient Memory Management: Features dynamic model loading, automatic unloading of inactive models, and prompt cache saving for fast session restoration.
• GGUF Support: Enables NPU inference for .GGUF models via specific llama.cpp forks.