microsoft/onnxruntime-genai

Generative AI with ONNX Runtime

What it solves

It provides a performant and flexible way to run generative AI models (specifically LLMs) locally on-device, handling the complex loop of generative AI including pre-processing, inference, and post-processing.

How it works

It implements the generative AI loop for ONNX models. This includes managing the KV cache, logits processing, search and sampling, and pre/post processing. It supports a wide range of hardware acceleration options (CPU, CUDA, DirectML, OpenVINO, QNN, WebGPU) and multiple model architectures like Llama, Phi, Mistral, and Gemma.

Who it’s for

Developers who want to integrate generative AI capabilities into their applications using ONNX Runtime, targeting various operating systems (Linux, Windows, Mac, Android) and hardware architectures (x86, x64, arm64).

Highlights

  • Multi-LoRA support
  • Constrained and continuous decoding
  • Broad support for model architectures including Llama, Phi, and Mistral
  • Cross-platform API support for Python, C#, C/C++, and Java
  • Extensive hardware acceleration options including CUDA and DirectML

Related

  • Project
    qualcomm/GenieX

    An on-device Gen AI inference runtime for Qualcomm Snapdragon devices that enables running LLMs and VLMs locally using the NPU, GPU, or CPU.

  • Project
    mlc-ai/web-llm

    A high-performance in-browser LLM inference engine that uses WebGPU to run language models locally without server-side processing.

  • Project
    google-ai-edge/LiteRT-LM

    A production-ready orchestration layer for running high-performance, cross-platform LLMs on-device using hardware acceleration.

  • Project
    nobodywho-ooo/nobodywho

    An on-device inference engine that enables running LLMs, speech-to-text, and text-to-speech locally and offline across mobile, desktop, and wearable platforms.

  • Project
    NVIDIA/TensorRT-LLM

    TensorRT‑LLM is NVIDIA’s open‑source library that compiles large language (and visual‑generation) models into ultra‑fast TensorRT inference engines. It provides custom CUDA kernels, quantization, sparse attention, speculative decoding, and multi‑GPU parallelism, delivering 2‑10× higher token throughput while keeping accuracy. The Python‑friendly runtime integrates with Triton, Kubernetes, and Jetson, and includes docs and examples for models like GPT‑OSS‑120B, Llama‑4, DeepSeek‑R1, and diffusion models.