vitoplantamura/OnnxStream

Lightweight inference library for ONNX files, written in C++. It can run Stable Diffusion XL 1.0 on a RPI Zero 2 (or in 298MB of RAM) but also Mistral 7B on desktops and servers. ARM, x86, WASM, RISC-V supported. Accelerated by XNNPACK. Python, C# and JS(WASM) bindings available.

What it solves

OnnxStream is a lightweight inference library designed to run large AI models—such as Stable Diffusion and LLMs—on hardware with extremely limited RAM (e.g., a Raspberry Pi Zero 2 with 512MB). It addresses the problem where standard machine learning frameworks prioritize speed and throughput over memory efficiency, often requiring gigabytes of RAM that low-end devices lack.

How it works

The library minimizes memory consumption by decoupling the inference engine from the weight loading process via a WeightsProvider. This allows weights to be streamed from disk or even an HTTP server instead of being loaded entirely into RAM. It employs several memory-saving techniques:

  • Attention Slicing: Prevents the creation of massive intermediate tensors during multi-head attention by splitting the query tensor into smaller chunks.
  • Quantization: Supports both dynamic (8-bit unsigned) and static (W8A8) quantization to reduce the precision and size of weights and activations.
  • Tiled Decoding: For large VAE decoders (like in SDXL), it splits the input tensor into overlapping tiles, decodes them separately, and blends them back together.
  • XNNPACK Integration: Uses XNNPACK for accelerated primitives like MatMul and Convolution.

Who it’s for

  • Developers targeting ultra-low-resource embedded devices (e.g., Raspberry Pi Zero).
  • Users wanting to run large generative models on hardware without dedicated GPUs or high RAM.
  • Engineers looking for a hackable, minimal C++ inference engine with Python and C# bindings.

Highlights

  • Extreme Memory Efficiency: Can consume up to 55x less memory than OnnxRuntime for certain models.
  • Broad Model Support: Capable of running Stable Diffusion 1.5, SDXL 1.0, SDXL Turbo, TinyLlama, Mistral 7B, YOLOv8, and OpenAI's Whisper.
  • Cross-Platform: Supports Linux, Mac, Windows, Termux, FreeBSD, and WebAssembly (browser-based execution).
  • Flexible Weight Loading: Weights can be provided via RAM, disk (with or without prefetching), or custom streaming providers.

Related

  • Dispatch
    Optimizing Stable Diffusion XL (SDXL) for Inference Speed and Memory

    Hugging Face explores several optimization techniques for Stable Diffusion XL (SDXL), demonstrating how to reduce memory usage from 28GB to as low as 11.47GB and decrease inference latency from 72.2 seconds to approximately 10.3 seconds.

  • Project
    huggingface/finetrainers

    A library for the accessible training and fine-tuning of diffusion models, focusing on memory-efficient text-to-video and text-to-image generation.

  • Project
    NVIDIA/TensorRT-LLM

    An open-source library from NVIDIA that optimizes inference for LLMs and visual generation models using specialized kernels and an efficient runtime to maximize throughput and latency on NVIDIA GPUs.

  • Project
    modelscope/DiffSynth-Studio

    An open-source Diffusion model engine that enables the inference and training of image, video, and audio generative models on consumer-grade GPUs through advanced VRAM management and quantization.

  • Project
    Windsander/ADI-Stable-Diffusion

    A C++ library and CLI tool that enables high-performance, Python-free inference for Stable Diffusion models using ONNXRuntime across multiple platforms.