SearchSavior/OpenArc

Inference engine for Intel devices. Serve LLMs, VLMs, Whisper, Kokoro-TTS, Embedding and Rerank models over OpenAI endpoints.

What it solves

OpenArc provides a simplified way to deploy and accelerate AI models on Intel hardware. It acts as an inference engine that allows users to run various model types locally and privately, removing the complexity of manually configuring OpenVINO acceleration for different AI tasks.

How it works

It leverages OpenVINO to serve models across CPU, GPU, and NPU devices. The engine provides OpenAI-compatible endpoints, allowing it to be integrated into existing AI workflows. It supports a wide range of model types, including LLMs, VLMs, Whisper, and various Qwen-based TTS, ASR, and embedding models.

Who it’s for

Developers and AI enthusiasts who want to run high-performance, local AI models specifically on Intel-based devices.

Highlights

  • Broad Model Support: Serves LLMs, VLMs, audio models (Whisper, Kokoro-TTS), and embedding/reranker models.
  • Intel Hardware Acceleration: Optimized for CPU, GPU, and NPU devices via OpenVINO.
  • OpenAI Compatibility: Provides standard endpoints for chat completions, audio transcriptions, and speech generation.
  • Inference Optimizations: Includes speculative decoding for LLMs and multi-GPU pipeline parallelism.
  • Performance Tracking: Built-in llama-bench style benchmarking and detailed request metrics like TTFT and throughput.

Related

  • Project
    openvinotoolkit/model_server

    A production-grade C++ inference server optimized for Intel hardware that serves Generative AI and classic deep learning models via OpenAI-compatible and KServe APIs.

  • Project
    openvinotoolkit/openvino

    An open-source toolkit for optimizing and deploying deep learning models across CPU, GPU, and NPU hardware to boost inference performance.

  • Project
    huggingface/optimum-intel

    An interface between Hugging Face libraries and OpenVINO that accelerates AI model inference and optimization on Intel architectures.

  • Project
    openvinotoolkit/openvino.genai

    A high-performance library for running Generative AI models locally on PCs and laptops, providing optimized pipelines for text, image, and speech models on Intel hardware.

  • Dispatch
    Optimizing and Deploying Hugging Face Models with Optimum-Intel and OpenVINO GenAI

    Hugging Face and Intel provide a streamlined workflow using Optimum-Intel and OpenVINO GenAI to optimize and deploy Transformers models on Intel hardware, specifically targeting edge and client-side C++ and Python environments.