robertknight/rten

ONNX neural network inference engine

What it solves

RTen is a machine learning runtime designed to let developers run models trained in Python (using frameworks like PyTorch) directly within Rust applications. It removes the need for a Python environment at runtime, providing a small, efficient, and end-to-end Rust implementation for model inference.

How it works

RTen loads models in the standard ONNX format or a custom .rten format for faster loading and support for very large models. It performs CPU-based inference, utilizing SIMD (AVX2, AVX-512, Arm Neon, and WebAssembly SIMD) and multi-threading to optimize performance. It also provides supporting libraries to handle the pre-processing and post-processing of data before and after it passes through the model.

Who it’s for

It is built for developers who want to deploy ML models in Rust environments or target WebAssembly (Wasm) for use in JavaScript applications, particularly those seeking a lightweight runtime without heavy external dependencies.

Highlights

  • End-to-end Rust: The entire project and its dependencies are written in Rust, simplifying build and deployment.
  • ONNX Support: Compatible with most standard ONNX operators and supports both float32 and quantized (int8/uint8) weights.
  • Cross-Platform: Runs on various platforms, including WebAssembly with SIMD support for browser-based inference.
  • Integrated Toolkit: Includes libraries for common pre- and post-processing tasks across different domains.

Related

  • Project
    pykeio/ort

    A Rust interface for hardware-accelerated inference and training of ONNX machine learning models, enabling efficient deployment across various hardware and platforms.

  • Project
    jamjamjon/usls

    A cross-platform Rust library powered by ONNX Runtime for efficient inference of vision and vision-language models under 1B parameters.

  • Project
    pegainfer-project/pegainfer

    A pure Rust and CUDA LLM inference engine that removes framework runtimes like PyTorch to provide faster startup times and a smaller memory footprint.

  • Project
    lablup/mlxcel

    A high-performance Rust-based inference runtime and server for LLMs and VLMs on Apple Silicon and NVIDIA CUDA, removing the need for Python environments during deployment.

  • Project
    tracel-ai/burn

    Burn is a unified tensor library and deep learning framework written in Rust that allows the same code to be used for both training and inference across diverse hardware.