google-ai-edge/litert-torch

What it solves

It enables developers to run PyTorch models on-device for Android, iOS, and IoT applications by converting them into the .tflite format. This removes the need for cloud-based inference, allowing models to run locally on CPU, GPU, and NPU hardware accelerators.

How it works

The library integrates with PyTorch using torch.export() and provides broad coverage of Core ATen operators to convert models into a .tflite flatbuffer. For Large Language Models (LLMs) and transformer-based architectures, it includes a specialized Generative API for authoring and quantizing mobile-optimized models that can be deployed via LiteRT-LM.

Who it’s for

It is designed for AI developers and mobile engineers who want to deploy PyTorch-trained models to edge devices while maintaining high performance across various hardware accelerators.

Highlights

• Broad Hardware Support: Initial support for CPU, GPU, and NPU.
• PyTorch Integration: Built on top of torch.export() for a seamless conversion workflow.
• Generative API: Specialized tools for authoring and quantizing transformer models for on-device LLM deployment.
• Cross-Platform: Targets Android, iOS, and IoT environments.