sonos/tract

Tiny, no-nonsense, self-contained, Tensorflow and ONNX inference

What it solves

tract is a neural-network inference engine designed to run AI models on a wide variety of hardware, from embedded ARM CPUs and WebAssembly in browsers to NVIDIA and Apple GPUs. It focuses on providing a tiny runtime footprint by allowing developers to translate models from formats like ONNX, TensorFlow Lite, and NNEF into a specialized intermediate representation (tract-OPL) so that heavy training-framework loaders and protobufs are not needed at runtime.

How it works

It loads models from formats such as ONNX, NNEF, and TFLite, optimizes them for the target hardware, and executes them. It uses a TypedModel IR (Intermediate Representation) that allows a model optimized for one platform to be moved to another. For real-time, low-latency workloads like wake-word detection or streaming speech recognition, it uses a process called "pulsification," which converts a network that normally processes full sequences into one that processes fixed-size pulses of data at each step.

Who it’s for

Developers building AI applications for embedded systems, browser-based AI, or workstations who need a high-performance, lightweight runtime for deploying models across different hardware targets.

Highlights

  • Multi-platform support: Runs on x86, ARM (including SVE), Apple Metal, NVIDIA CUDA, and WebAssembly.
  • Broad model support: Compatible with ONNX, NNEF, and TFLite, with a direct PyTorch-to-NNEF path via torch-to-nnef.
  • Small runtime footprint: Uses tract-OPL to ship only the necessary runtime components, avoiding bulky loaders.
  • Streaming inference: First-class support for pulsified inference for real-time, low-latency audio/speech workloads.
  • Language bindings: Available in both Rust and Python.

Related

  • Project
    google/XNNPACK

    XNNPACK is a highly optimized library of low-level performance primitives used to accelerate neural network inference across ARM, x86, WebAssembly, and RISC-V platforms.

  • Project
    microsoft/onnxruntime

    A cross-platform machine learning accelerator that optimizes inference and training for models from various frameworks like PyTorch and TensorFlow.

  • Project
    pykeio/ort

    A Rust interface for hardware-accelerated inference and training of ONNX machine learning models, enabling efficient deployment across various hardware and platforms.

  • Project
    Samsung/ONE

    ONE is a high-performance on-device neural network inference framework that enables AI models from TensorFlow and PyTorch to run efficiently across various hardware accelerators.

  • Project
    mil-tokyo/webdnn

    A browser-based neural network inference engine that runs ONNX models directly in the web browser using WebGPU, WebGL, and WebAssembly.