pykeio/ort

Fast ML inference & training for ONNX models in Rust

What it solves

ort provides a high-performance Rust interface for running machine learning models in the ONNX format. It simplifies the process of deploying models trained in frameworks like PyTorch, TensorFlow, Keras, scikit-learn, or PaddlePaddle to either on-device environments or data centers.

How it works

It acts primarily as a wrapper for Microsoft's ONNX Runtime library, allowing Rust developers to leverage hardware acceleration across a wide variety of accelerators. It also supports other pure-Rust runtimes as backends.

Who it’s for

Rust developers who need to perform hardware-accelerated inference and training on ONNX models across different platforms, from edge devices to servers.

Highlights

  • Broad Framework Support: Supports models from PyTorch, TensorFlow, Keras, scikit-learn, and PaddlePaddle.
  • Hardware Acceleration: Compatible with almost any hardware accelerator via ONNX Runtime.
  • Versatile Deployment: Light enough for on-device use while remaining powerful enough for data center deployment.
  • Wide Adoption: Used by projects like Hugging Face's Text Embeddings Inference (TEI) and Google's Magika.

Related

  • Project
    robertknight/rten

    A machine learning runtime written in Rust that enables the execution of ONNX models trained in Python on various platforms, including WebAssembly.

  • Project
    microsoft/onnxruntime

    A cross-platform machine learning accelerator that optimizes inference and training for models from various frameworks like PyTorch and TensorFlow.

  • Project
    jamjamjon/usls

    A cross-platform Rust library powered by ONNX Runtime for efficient inference of vision and vision-language models under 1B parameters.

  • Project
    Anush008/fastembed-rs

    A Rust library for generating text, image, and sparse vector embeddings and reranking locally using ONNX Runtime.

  • Dispatch
    Hugging Face Optimum and ONNX Runtime Training Integration

    Hugging Face and Microsoft have integrated ONNX Runtime into the Optimum library to reduce training times for transformer-based models by 35% or more.