SearchSavior/OpenArc

Inference engine for Intel devices. Serve LLMs, VLMs, Whisper, Kokoro-TTS, Embedding and Rerank models over OpenAI endpoints.

What it solves

OpenArc simplifies the deployment and acceleration of various AI models on Intel hardware. It allows users to host multiple types of models—such as LLMs, vision-language models (VLMs), speech-to-text, and text-to-speech—on a single server simultaneously, providing a local and private alternative to cloud-based AI services.

How it works

It acts as an inference engine powered by OpenVINO, enabling hardware acceleration across Intel CPUs, GPUs, and NPUs. OpenArc exposes these models through OpenAI-compatible endpoints, making it easy to integrate with existing tools. It features an asynchronous multi-engine architecture that supports model concurrency, speculative decoding for LLMs, and multi-GPU pipeline parallelism.

Who it’s for

Developers and users who want to run high-performance, private AI models locally on Intel-based devices.

Highlights

  • Broad Model Support: Serves LLMs, VLMs, Whisper, Kokoro-TTS, Qwen-TTS, Qwen-ASR, embeddings, and rerankers.
  • Hardware Flexibility: Supports CPU, GPU, and NPU devices, including hybrid device offloading.
  • OpenAI Compatibility: Provides standard endpoints for chat, completions, audio transcriptions, and speech.
  • Performance Tooling: Includes llama-bench style benchmarking with an automatic SQLite database and detailed request metrics (e.g., TTFT, throughput).
  • Advanced Inference: Supports speculative decoding and streaming cancellation for LLMs and VLMs.

Related

  • Project
    openvinotoolkit/openvino

    An open-source toolkit for optimizing and deploying deep learning models across CPU, GPU, and NPU hardware to boost inference performance.

  • Project
    openvinotoolkit/model_server

    OpenVINO Model Server (OVMS) is a C++ inference server that serves LLMs, vision‑language models, and classic deep‑learning models via OpenAI‑compatible REST or KServe gRPC APIs. Optimized for Intel CPUs, GPUs and NPUs, it provides features like continuous batching, streaming, model versioning, hot‑reload, and Prometheus metrics. Deployable via Docker, bare‑metal binaries, or Kubernetes, it supports many model formats (TensorFlow, ONNX, OpenVINO IR, GGUF, etc.) and integrates with standard client libraries.

  • Project
    huggingface/optimum-intel

    An interface between Hugging Face libraries and OpenVINO that accelerates AI model inference and optimization on Intel architectures.

  • Project
    openvinotoolkit/openvino.genai

    A lightweight library of optimized generative AI model pipelines for high-performance local execution on CPU, GPU, and NPU via the OpenVINO Runtime.

  • Project
    triton-inference-server/server

    An open-source inference serving software that streamlines the deployment of AI models from multiple frameworks across cloud, data center, edge, and embedded devices.