google-ai-edge/LiteRT

LiteRT, successor to TensorFlow Lite. is Google's On-device framework for high-performance ML & GenAI deployment on edge platforms, via efficient conversion, runtime, and optimization

📚 What is LiteRT?

LiteRT is Google’s on‑device runtime for running machine‑learning and generative‑AI models on edge hardware (phones, laptops, browsers, IoT devices, etc.). It is positioned as the successor to TensorFlow Lite, offering a unified C++/Kotlin/JS API that automatically picks the best accelerator (CPU, GPU, NPU, WebGPU, …) and supports asynchronous execution.

🚀 Core capabilities

Capability What it means for you
Compiled Model API Load a model file (.tflite or .litertlm) and run inference with a single call. No need to manually attach delegates – LiteRT picks the optimal accelerator for you.
Unified NPU support One API works across NPUs from Google, Qualcomm, MediaTek, Intel, Broadcom, etc., so you can write code once and run on many devices.
GPU acceleration via ML Drift Uses a new buffer‑interop layer to minimise latency on GPUs, which is especially useful for generative‑AI workloads.
Web inference Through LiteRT.js you can run models in the browser using WebGPU and WebAssembly, giving client‑side AI without a server.
C++ Tensor API A lightweight, tensor‑centric library for building and manipulating graphs directly in C++.
CLI tooling litert command‑line utility helps coding‑agent workflows (model conversion, quantisation, benchmarking, etc.).

🛠 How to get started

  1. Install the CLI (requires Python 3.13):
    uv venv --clear --python=3.13 --seed
source .venv/bin/activate
uv pip install litert-cli-nightly
litert --help
  2. Choose a platform – Android, iOS, Linux, macOS, Windows, Web, or IoT. The README lists the supported accelerators for each.
  3. Convert your model
    • PyTorch → LiteRT: use the LiteRT Torch Converter (litert-torch).
    • For generative models (LLMs, diffusion) export to .litertlm via the LiteRT‑LM library.
  4. Run inference with the Compiled Model API in the language of your app (C++, Kotlin, Swift, JavaScript, Python).
  5. Optional – explore sample apps (image segmentation, speech‑recognition) in the litert-samples repo for ready‑made code.

📦 Ecosystem & related projects

Project Role
LiteRT‑Torch Converter & generative API for PyTorch models.
LiteRT‑LM Runtime specialised for on‑device LLM inference.
LiteRT.js Web‑focused runtime (WebGPU + WASM).
AI‑Edge Quantizer Advanced quantisation before feeding a model to LiteRT.
MediaPipe Complementary framework for building media‑centric pipelines that can consume LiteRT models.
XNNPACK Low‑level CPU operator library used under the hood.

🎯 Who should use LiteRT?

  • Mobile developers who want a single API that works on Android and iOS and automatically leverages GPUs/NPUs.
  • Edge‑AI engineers targeting Linux, Windows, or IoT boards (Raspberry Pi) and need high‑throughput inference.
  • Web developers looking to run ML models client‑side with WebGPU.
  • Generative‑AI practitioners who want to run quantised LLMs or diffusion models locally without a cloud backend.
  • Tool builders / coding agents that need a CLI for model conversion, benchmarking, and deployment automation.

📚 Where to learn more

🛡 License & contribution

LiteRT is released under the Apache‑2.0 license. Contributions are welcomed via the usual pull‑request workflow; see CONTRIBUTING.md for details.

Related

  • Project
    google-ai-edge/litert-samples

    A collection of official samples, model recipes, and deployment utilities for LiteRT and LiteRT-LM to help developers deploy high-performance ML models on-device.

  • Project
    google-ai-edge/LiteRT-LM

    A production-ready orchestration layer for running high-performance, cross-platform LLMs on-device using hardware acceleration.

  • Project
    microsoft/onnxruntime-genai

    A high-performance API for running generative AI models on-device using ONNX Runtime, handling the full generative loop from pre-processing to sampling.

  • Project
    NightMean/OlliteRT

    An Android application that turns a mobile phone into a local, OpenAI-compatible LLM server using the LiteRT-LM runtime.

  • Project
    NVIDIA/TensorRT-LLM

    TensorRT‑LLM is NVIDIA’s open‑source library that compiles large language (and visual‑generation) models into ultra‑fast TensorRT inference engines. It provides custom CUDA kernels, quantization, sparse attention, speculative decoding, and multi‑GPU parallelism, delivering 2‑10× higher token throughput while keeping accuracy. The Python‑friendly runtime integrates with Triton, Kubernetes, and Jetson, and includes docs and examples for models like GPT‑OSS‑120B, Llama‑4, DeepSeek‑R1, and diffusion models.