openvinotoolkit/model_server

A scalable inference server for models optimized with OpenVINO™

What it solves

OpenVINO Model Server (OVMS) provides a production-grade way to deploy and serve ML models over network APIs. It eliminates the need to build custom serving infrastructure for both Generative AI (LLMs, VLMs, image and audio generation) and classic deep learning models (classification, object detection, OCR).

How it works

Written in C++, OVMS acts as an inference server that exposes models via standard APIs. It leverages the OpenVINO toolkit to optimize model execution on Intel hardware, including CPUs, GPUs, and NPUs. It supports multiple model formats (TensorFlow, ONNX, PaddlePaddle, OpenVINO IR) and can pull models from various repositories such as HuggingFace Hub, S3, GCS, and Azure Blob.

Who it’s for

Developers and ML engineers who need to deploy models on Intel-based infrastructure (Docker, Kubernetes, bare metal) and want standard API compatibility (OpenAI-compatible for GenAI, KServe for classic models).

Highlights

  • Broad GenAI Support: Supports LLM text generation with continuous batching, streaming, and speculative decoding, as well as VLMs, embeddings, and reranking.
  • Hardware Acceleration: Optimized specifically for Intel CPU, integrated/discrete GPU, and NPU.
  • Flexible Deployment: Runs on Linux and Windows via Docker or binary packages.
  • Enterprise Features: Includes model versioning, hot-reload, and Prometheus-compatible metrics for production monitoring.
  • Multi-modal Capabilities: Handles text, image, audio, and speech recognition/TTS via OpenAI-compatible APIs.

Related

  • Project
    SearchSavior/OpenArc

    An OpenVINO-powered inference engine for Intel devices that serves LLMs, VLMs, and audio models via OpenAI-compatible endpoints.

  • Project
    openvinotoolkit/openvino

    An open-source toolkit for optimizing and deploying deep learning models across CPU, GPU, and NPU hardware to boost inference performance.

  • Project
    openvinotoolkit/openvino.genai

    A high-performance library for running Generative AI models locally on PCs and laptops, providing optimized pipelines for text, image, and speech models on Intel hardware.

  • Project
    huggingface/optimum-intel

    An interface between Hugging Face libraries and OpenVINO that accelerates AI model inference and optimization on Intel architectures.

  • Dispatch
    Optimizing and Deploying Hugging Face Models with Optimum-Intel and OpenVINO GenAI

    Hugging Face and Intel provide a streamlined workflow using Optimum-Intel and OpenVINO GenAI to optimize and deploy Transformers models on Intel hardware, specifically targeting edge and client-side C++ and Python environments.