pykeio/ort
Fast ML inference & training for ONNX models in Rust
What it solves
ort provides a high-performance Rust interface for running machine learning models in the ONNX format. It solves the problem of deploying models trained in various frameworks (like PyTorch, TensorFlow, or scikit-learn) into production environments—either on-device or in the datacenter—without needing the original training framework's heavy runtime.
How it works
It acts primarily as a wrapper for Microsoft's ONNX Runtime library, allowing Rust developers to leverage hardware acceleration across a wide range of hardware accelerators. It also supports other pure-Rust runtimes as backends.
Who it’s for
Rust developers who need to integrate machine learning model inference and training into their applications, specifically those targeting high-performance, on-device, or cross-platform deployment.
Highlights
- Broad Framework Support: Supports models exported from PyTorch, TensorFlow, Keras, scikit-learn, and PaddlePaddle.
- Hardware Acceleration: Compatible with almost any hardware accelerator via ONNX Runtime.
- Lightweight: Designed to be efficient enough for deployment on end-user devices.
- Wide Adoption: Used by projects like Hugging Face's Text Embeddings Inference (TEI) and Google's Magika.