openvinotoolkit/openvino.genai

Run Generative AI models with simple C++/Python API and using OpenVINO Runtime

What it solves

OpenVINO GenAI provides a lightweight, high-performance library for running popular Generative AI models locally on PCs and laptops. It eliminates the need for external dependencies by integrating core functionality like tokenization directly into the library, reducing resource consumption for local execution.

How it works

Built on top of the OpenVINO Runtime, the library provides C++, Python, and Node.js APIs to execute various generative pipelines. It supports inference on CPU, GPU, and NPU hardware. The library includes specialized optimization techniques such as speculative decoding, KVCache token eviction, and sparse attention to accelerate generation and reduce memory usage.

Who it’s for

Developers building local AI applications who need to run LLMs, image generators, and speech models on Intel hardware without complex dependency chains.

Highlights

  • Multi-modal support: Covers text generation (LLMs), visual processing (VLMs), image generation (Diffusers), speech recognition (Whisper), speech generation (SpeechT5), and RAG components like text embedding and reranking.
  • LoRA adapter support: Allows loading multiple adapters per model and mixing them using alpha blending.
  • Advanced optimizations: Implements speculative decoding, sparse attention (Tri-shape and XAttention), and continuous batching with prefix caching for serving scenarios.
  • Hardware acceleration: Optimized for seamless execution across CPU, GPU, and NPU.

Related

  • Project
    openvinotoolkit/openvino

    An open-source toolkit for optimizing and deploying deep learning models across CPU, GPU, and NPU hardware to boost inference performance.

  • Project
    SearchSavior/OpenArc

    An OpenVINO-powered inference engine for Intel devices that serves LLMs, VLMs, and audio models via OpenAI-compatible endpoints.

  • Project
    openvinotoolkit/model_server

    A production-grade C++ inference server optimized for Intel hardware that serves Generative AI and classic deep learning models via OpenAI-compatible and KServe APIs.

  • Project
    huggingface/optimum-intel

    An interface between Hugging Face libraries and OpenVINO that accelerates AI model inference and optimization on Intel architectures.

  • Dispatch
    Optimizing and Deploying Hugging Face Models with Optimum-Intel and OpenVINO GenAI

    Hugging Face and Intel provide a streamlined workflow using Optimum-Intel and OpenVINO GenAI to optimize and deploy Transformers models on Intel hardware, specifically targeting edge and client-side C++ and Python environments.