microsoft/onnxruntime-genai

What it solves

It provides a high-performance, flexible API for running generative AI models (specifically LLMs) locally on-device, removing the need for complex manual implementation of the generative loop.

How it works

The project implements the full generative AI loop for ONNX models. This includes handling pre-processing, performing inference via ONNX Runtime, managing the KV cache, processing logits, and executing search and sampling strategies. It also supports grammar specification for tool calling.

Who it’s for

Developers who want to deploy generative AI models across various platforms (Windows, Linux, Mac, Android) and hardware accelerators (CPU, CUDA, DirectML, OpenVINO, QNN, WebGPU) using the ONNX format.

Highlights

• Broad Model Support: Supports a wide range of architectures including Llama, Phi, Mistral, Gemma, and Qwen.
• Cross-Platform: Works across multiple operating systems and provides APIs for Python, C#, C/C++, and Java.
• Hardware Acceleration: Leverages various backends like CUDA, DirectML, and WebGPU for optimized performance.
• Advanced Generation Features: Includes support for Multi-LoRA, continuous decoding, and constrained decoding.