robertknight/rten

ONNX neural network inference engine

What it solves

RTen is a machine learning runtime designed to run models trained in Python (such as those from PyTorch) within Rust applications. It eliminates the need for complex C++ dependencies by providing an end-to-end Rust implementation, simplifying the build and deployment process across various platforms.

How it works

RTen supports the ONNX format, allowing users to export models from Python frameworks and load them into a Rust environment. It performs CPU-based inference using SIMD (AVX2, AVX-512, Arm Neon, and WebAssembly SIMD) to optimize performance. The engine supports single-precision float (f32) and quantized int8 weights, with partial support for half-precision (f16) weights via upconversion to f32.

Who it’s for

Developers who want to integrate machine learning models into Rust applications or deploy models to the web via WebAssembly, without relying on heavy external C++ runtimes.

Highlights

  • End-to-end Rust implementation for simplified deployment.
  • Support for most standard ONNX operators and some ONNX Runtime "contrib" ops.
  • CPU-only inference with multi-threading and SIMD acceleration.
  • First-class support for WebAssembly, enabling ML inference in JavaScript applications.
  • Includes supporting libraries for common pre-processing and post-processing tasks.

Related

  • Project
    pykeio/ort

    A Rust interface for hardware-accelerated inference and training of ONNX machine learning models, enabling efficient deployment across various hardware.

  • Project
    jamjamjon/usls

    A cross-platform Rust library powered by ONNX Runtime for efficient inference of vision and vision-language models under 1B parameters.

  • Project
    mil-tokyo/webdnn

    A browser-based neural network inference engine that runs ONNX models directly in the web browser using WebGPU, WebGL, and WebAssembly.

  • Project
    lablup/mlxcel

    A high-performance native Rust runtime for LLM, VLM, and audio inference on Apple Silicon, CUDA, and ROCm, removing Python from the production request path.

  • Project
    guoqingbao/xinfer

    A blazing-fast LLM inference engine written in pure Rust that eliminates Python/PyTorch dependencies to enable high-throughput, production-ready model serving on consumer and professional GPUs.