google-ai-edge/LiteRT-LM

LiteRT-LM is Google's production-ready, high-performance, open-source inference framework for deploying Large Language Models on edge devices.

What it solves

LiteRT-LM is a production-ready orchestration layer designed to run large language models (LLMs) on-device. It addresses the challenge of executing high-performance AI across diverse hardware platforms, including mobile devices, wearables, and desktop environments, without relying on cloud infrastructure.

How it works

It provides a cross-platform execution environment that leverages hardware acceleration via GPUs and NPUs to maximize performance. The system supports a wide range of models (such as Gemma, Llama, Phi-4, and Qwen) and provides APIs for multiple languages including Python, Kotlin, Swift, JavaScript, and C++.

Who it’s for

Developers building on-device GenAI experiences for Android, iOS, Web, Desktop, and IoT devices (e.g., Raspberry Pi), as well as those integrating AI into products like Chrome and Chromebook Plus.

Highlights

  • Cross-Platform Support: Works across Android, iOS, Web, Desktop, and IoT.
  • Hardware Acceleration: Optimized for GPU and NPU accelerators for peak performance.
  • Multi-Modality: Supports vision and audio inputs alongside text.
  • Tool Use: Includes function calling support for agentic workflows.
  • Broad Model Support: Compatible with popular models like Gemma, Llama, Phi-4, and Qwen.
  • Developer Tooling: Includes a CLI for fast model imports and an OpenAI-compatible embeddings endpoint.

Related

  • Project
    NightMean/OlliteRT

    An Android application that turns a mobile phone into a local, OpenAI-compatible LLM server using the LiteRT-LM runtime.

  • Project
    Lightning-AI/litgpt

    A high-performance framework for pretraining, finetuning, and deploying 20+ LLMs with from-scratch implementations and no abstraction layers.

  • Project
    google-ai-edge/gallery

    A mobile and macOS application for running open-source LLMs and multimodal AI models fully on-device, providing a private and offline experience for chat, media search, and audio transcription.

  • Project
    google-ai-edge/LiteRT

    LiteRT is Google’s high‑performance on‑device runtime for running ML and generative‑AI models across phones, laptops, browsers, and IoT devices. It supersedes TensorFlow Lite with a unified Compiled Model API that automatically selects the best accelerator (CPU, GPU, NPU, WebGPU). The ecosystem includes converters for PyTorch, a CLI for coding‑agent workflows, a Web‑focused LiteRT.js, and a library for on‑device LLM inference. Supported platforms span Android, iOS, Linux, macOS, Windows, Web, and Raspberry Pi, with broad hardware acceleration support.

  • Project
    google-ai-edge/litert-samples

    A collection of official samples, model recipes, and deployment utilities for LiteRT and LiteRT-LM to help developers deploy high-performance ML models on-device.