ort: a Rust interface for hardware-accelerated ONNX model inference and training

What it solves

ort provides a high-performance Rust interface for running machine learning models in the ONNX format. It simplifies the deployment of models created in frameworks like PyTorch, TensorFlow, Keras, scikit-learn, and PaddlePaddle, allowing them to be executed efficiently on a variety of hardware accelerators across different environments, from data centers to end-user devices.

How it works

It acts as a wrapper for Microsoft's ONNX Runtime library, while also supporting other pure-Rust runtimes. This allows Rust developers to leverage hardware acceleration for both inference and training of ONNX models without needing to write complex low-level bindings.

Who it’s for

Rust developers who need to deploy machine learning models on-device or in the cloud with high performance and hardware acceleration.

Highlights

• Broad Framework Support: Supports models from PyTorch, TensorFlow, Keras, scikit-learn, and PaddlePaddle.
• Hardware Acceleration: Compatible with almost any hardware accelerator via ONNX Runtime.
• Versatile Deployment: Light enough for on-device use, yet powerful enough for datacenter deployment.
• Wide Adoption: Used by projects like Hugging Face's Text Embeddings Inference (TEI) and Google's Magika.