google-ai-edge/LiteRT-LM

LiteRT-LM is Google's production-ready, high-performance, open-source inference framework for deploying Large Language Models on edge devices.

What it solves

LiteRT-LM is a production-ready orchestration layer designed to run Large Language Models (LLMs) on-device. It addresses the challenge of achieving high-performance, cross-platform execution of AI models on edge devices like smartphones, laptops, and IoT hardware without relying on cloud infrastructure.

How it works

It acts as an orchestration layer for LiteRT, providing a unified API for executing models such as Gemma, Llama, Phi-4, and Qwen. It leverages hardware acceleration via GPUs and NPUs to maximize performance and and supports various backends. It also includes a CLI for quick testing and provides language-specific APIs for Python, Kotlin, Swift, JavaScript, and C++.

Who it’s for

Developers building on-device GenAI experiences for Android, iOS, Web, and Desktop platforms, as well as those targeting IoT devices like Raspberry Pi.

Highlights

  • Cross-Platform Support: Works across Android, iOS, Web, Desktop, and IoT.
  • Hardware Acceleration: Utilizes GPU and NPU accelerators for peak performance.
  • Multi-Modality: Supports vision and audio inputs.
  • Tool Use: Includes function calling support for agentic workflows.
  • Broad Model Support: Compatible with models like Gemma, Llama, Phi-4, and Qwen.
  • Multi-Token Prediction: Supports MTP drafters to increase inference speed.

Related

  • Project
    google-ai-edge/LiteRT

    Google's high-performance on-device runtime for deploying ML and Generative AI models across mobile, web, and IoT platforms with advanced hardware acceleration.

  • Project
    google-ai-edge/litert-samples

    A collection of official and community samples, model recipes, and deployment tools for LiteRT and LiteRT-LM to enable high-performance on-device machine learning.

  • Project
    google-ai-edge/gallery

    A mobile and macOS application for running open-source LLMs locally on-device, providing a private and offline environment to experience and evaluate generative AI.

  • Project
    dineshsoudagar/local-llms-on-android

    An Android application that enables fully offline, private LLM chat with support for voice, image, and camera input using ONNX and LiteRT backends.

  • Project
    Lightning-AI/litgpt

    LitGPT is a high-performance framework for pretraining, finetuning, and deploying over 20 popular LLMs with minimal abstraction for maximum control and scale.