sonos/tract

Tiny, no-nonsense, self-contained, Tensorflow and ONNX inference

What it solves

tract is a neural-network inference engine designed to run AI models on a wide variety of hardware with a minimal runtime footprint. It solves the problem of deploying models across diverse environments—from embedded ARM CPUs and GPUs (NVIDIA/Apple) to web browsers via WebAssembly—without needing to ship heavy training frameworks or large loaders.

How it works

tract loads models from formats like ONNX, NNEF, and TFLite, then optimizes and compiles them for a specific runtime. It uses a specialized intermediate representation called tract-OPL (based on NNEF) to allow a "translate-once, ship-tiny-runtime" workflow, where models are converted to a compact format at build time so the final deployment only requires a small set of core libraries.

It also features "pulsification," which transforms models that were trained on full sequences into versions that can process fixed-size "pulses" of data, enabling low-latency, real-time streaming inference for tasks like wake-word detection.

Who it’s for

It is built for developers deploying AI models to production environments where resource efficiency is critical, such as embedded devices, browser-based applications, and workstations across Linux, macOS, and Windows.

Highlights

  • Multi-platform support: Runs on x86, ARM (including SVE), Apple Metal, NVIDIA CUDA, and WebAssembly.
  • Broad model compatibility: Supports ONNX, NNEF, TFLite, and legacy TensorFlow 1 frozen graphs.
  • Streaming inference: First-class support for pulsified networks to enable real-time, low-latency processing.
  • Minimal footprint: Optimized for small runtime sizes by separating the translation process from the execution engine.

Related

  • Project
    google/XNNPACK

    XNNPACK is a highly optimized library of low-level performance primitives used to accelerate neural network inference across ARM, x86, WebAssembly, and RISC-V platforms.

  • Project
    microsoft/onnxruntime

    A cross-platform machine learning accelerator that optimizes inference and training for models from various frameworks like PyTorch and TensorFlow.

  • Project
    pykeio/ort

    A Rust interface for hardware-accelerated inference and training of ONNX machine learning models, enabling efficient deployment across various hardware.

  • Project
    triton-inference-server/server

    An open-source inference serving software that streamlines the deployment of AI models from multiple frameworks across cloud, data center, edge, and embedded devices.

  • Project
    mil-tokyo/webdnn

    A browser-based neural network inference engine that runs ONNX models directly in the web browser using WebGPU, WebGL, and WebAssembly.