netdur/llama_cpp_dart

dart binding for llama.cpp

What it solves

llama_cpp_dart provides a high-performance Dart FFI binding for llama.cpp, specifically optimized for integrating Large Language Models (LLMs) directly into Flutter mobile applications on iOS and Android. It removes the complexity of manually managing native build toolchains (like CMake or the Android NDK) by providing pre-built binaries for various mobile platforms, including specialized support for Snapdragon Hexagon NPUs.

How it works

The project acts as a bridge between Dart and the C++ llama.cpp library. It uses a worker isolate to run inference off-thread, ensuring that the user interface (UI) remains responsive during token generation. It supports streaming output via Dart Streams and handles multimodal inputs (images and audio) through the mtmd library. To manage memory on constrained mobile devices, it implements KV-cache quantization, allowing longer contexts to fit within limited RAM.

Who it’s for

Flutter developers who want to embed local, on-device AI capabilities—such as chat, vision, and audio processing—into their mobile apps without relying on cloud APIs.

Highlights

  • Mobile-First Acceleration: Support for Apple Metal, Android CPU, and Snapdragon Hexagon NPU/OpenCL.
  • Off-Thread Inference: Uses a dedicated LlamaEngine worker isolate to prevent UI blocking.
  • Multimodal Support: Ability to process image and audio bitmaps directly within the model.
  • Memory Optimization: KV-cache quantization (e.g., q8_0, iq4_nl) to reduce RAM usage for longer contexts.
  • State Persistence: Save and restore conversation history and KV-caches to a single metadata-validated file.
  • Advanced LLM Features: Includes speculative decoding, context shifting (auto-shift), and integrated Jinja chat templates.

Related

  • Project
    abetlen/llama-cpp-python

    Python bindings for llama.cpp that enable local LLM inference with an OpenAI-compatible API and support for multiple hardware acceleration backends.

  • Project
    orailnoor/cross-platform-llm-client

    A cross-platform AI chat client that unifies local on-device LLM inference via llama.cpp with cloud API access for OpenAI, Anthropic, and Google Gemini.

  • Project
    Vali-98/ChatterUI

    A native mobile frontend for LLMs that allows users to run models on-device using llama.cpp or connect to various commercial and open-source APIs.

  • Dispatch
    llama.cpp and llama.app: Local LLM Inference and Ecosystem

    llama.cpp and its official home llama.app provide high-performance, hardware-agnostic local LLM inference, now featuring a simplified 'llama serve' command and integration with local coding agents like Pi.

  • Project
    withcatai/node-llama-cpp

    A Node.js library that provides pre-built bindings for llama.cpp, enabling developers to run LLMs locally with automatic hardware acceleration and TypeScript support.