pytorch/executorch

ExecuTorch – On‑device AI inference built on PyTorch

What it is – ExecuTorch is Meta’s open‑source runtime that lets you take a regular PyTorch model, export it with torch.export(), compile it ahead‑of‑time for a target edge device, and run it with a tiny C++/Python/Swift/Kotlin runtime. It is designed for privacy‑preserving, low‑latency inference on anything from smartphones to micro‑controllers.

Why it matters

• Native PyTorch workflow – No need to convert to ONNX, TFLite, etc.; you stay in the familiar PyTorch ecosystem.
• Tiny footprint – The base runtime is ~50 KB, making it suitable for MCUs.
• Multiple back‑ends – One exported .pte file can run on 12+ hardware back‑ends (XNNPACK, CoreML, Qualcomm QNN, Vulkan, ARM Ethos‑U, etc.) by swapping a single line of code.
• Production‑proven – Powers on‑device AI in Meta’s Instagram, WhatsApp, Quest 3, Ray‑Ban Smart Glasses and more.

How it works

1. Export – Capture the model graph with torch.export().
2. Compile – Apply quantization, optimizations and partition the graph to hardware‑specific sub‑graphs, producing a .pte file.
3. Execute – Load the file with the lightweight ExecuTorch runtime (available via Python pybind, C++, Swift, or Kotlin) and run inference.

Quick start

pip install executorch

import torch
from executorch.exir import to_edge_transform_and_lower
from executorch.backends.xnnpack.partition.xnnpack_partitioner import XnnpackPartitioner

model = MyModel().eval()
example = (torch.randn(1,3,224,224),)
exported = torch.export.export(model, example)
program = to_edge_transform_and_lower(
    exported,
    partitioner=[XnnpackPartitioner()]
).to_executorch()
open('model.pte','wb').write(program.buffer)

Run locally with the Python runtime or on a device with the C++/Swift/Kotlin APIs shown in the README.

LLM support – ExecuTorch can export and run Llama‑style models (e.g., Llama 3.2) using the export_llm script or HuggingFace’s Optimum‑ExecuTorch integration. A dedicated LLM runner API handles text generation and multimodal inputs.

Key features

• Built‑in 8‑bit/4‑bit quantization via torchao
• Memory‑planning and AOT allocation to keep RAM usage low
• Profiling tools (ETDump, ETRecord)
• Selective operator stripping to shrink binaries
• Custom operator extensions and dynamic‑shape support

Where it runs

Platform	Back‑ends
Android	XNNPACK, Vulkan, Qualcomm, MediaTek, Samsung Exynos
iOS	XNNPACK, CoreML (Neural Engine)
Linux/Windows	XNNPACK, OpenVINO, CUDA (exp.)
macOS	XNNPACK, Metal (exp.), MLX (exp.)
Embedded/MCU	XNNPACK, ARM Ethos‑U, NXP, Cadence DSP

Resources

• Full docs: https://docs.pytorch.org/executorch/main/index.html
• Example models (Llama, Qwen, Phi‑4, Llava, Whisper, YOLO, etc.) in the examples/ folder
• Community: Discord, GitHub Discussions, Issues, and a BSD‑style license.

Bottom line – If you already develop models in PyTorch and need to ship them to edge devices without rewriting code or dealing with proprietary formats, ExecuTorch provides a unified, production‑grade path from research to on‑device deployment.