google-ai-edge/LiteRT

LiteRT, successor to TensorFlow Lite. is Google's On-device framework for high-performance ML & GenAI deployment on edge platforms, via efficient conversion, runtime, and optimization

What it solves

LiteRT provides a high-performance runtime for deploying machine learning and Generative AI models on edge devices. It simplifies the process of running inference on mobile, web, and IoT platforms while maximizing the use of available hardware acceleration (CPU, GPU, and NPU).

How it works

LiteRT acts as an execution engine that runs optimized models converted from frameworks like PyTorch, TensorFlow, and Jax. It utilizes a Compiled Model API to automate accelerator selection and handle asynchronous execution. For GenAI, it uses specialized tools like LiteRT-LM for LLMs and LiteRT.js for browser-based inference via WebGPU and WASM.

Who it’s for

Developers building on-device AI applications for Android, iOS, Linux, macOS, Windows, and the Web, as well as those looking to migrate from TensorFlow Lite.

Highlights

  • Unified NPU Acceleration: A single API to access NPUs from various chipset providers.
  • Cross-Platform Support: Compatible with Android, iOS, Linux, macOS, Windows, Web, and IoT (Raspberry Pi).
  • GenAI Ready: Dedicated support for deploying quantized LLMs and diffusion models on-device.
  • Compiled Model API: Streamlines development by removing the need for manual delegate creation and improving I/O buffer handling.

Related

  • Project
    google-ai-edge/LiteRT-LM

    A production-ready orchestration layer for running high-performance LLMs on-device across mobile, web, and desktop platforms using LiteRT.

  • Project
    google-ai-edge/litert-samples

    A collection of official and community samples, model recipes, and deployment tools for LiteRT and LiteRT-LM to enable high-performance on-device machine learning.

  • Project
    microsoft/onnxruntime-genai

    A high-performance API for running generative AI models on-device using ONNX Runtime, handling the complete generative loop from pre-processing to sampling.

  • Project
    NVIDIA/TensorRT-LLM

    An open-source library from NVIDIA that optimizes inference for LLMs and visual generation models using specialized kernels and an efficient runtime to maximize throughput and latency on NVIDIA GPUs.

  • Project
    tile-ai/TileRT

    A tile-based runtime engine for ultra-low-latency LLM inference that minimizes time per output token for massive models on NVIDIA B200 GPUs.