openvinotoolkit/model_server

OpenVINO™ Model Server (OVMS)

What it is – A production‑grade C++ inference server that lets you expose machine‑learning models (LLMs, vision‑language models, classic vision/classification models, etc.) over standard network APIs. It is built on the OpenVINO toolkit and is tuned for Intel CPUs, GPUs and NPUs.

Why it matters – It lets you run generative‑AI workloads (text generation, embeddings, image generation, speech) and traditional deep‑learning inference behind an OpenAI‑compatible REST API or the KServe gRPC API, with hardware acceleration and features such as continuous batching, streaming, model versioning and hot‑reload.

---

Core capabilities

Category	Highlights
Generative AI	OpenAI‑compatible endpoints for chat/completions, embeddings, image generation, audio (ASR/TTS). Supports continuous batching, speculative decoding, structured output, and GGUF model format.
Classic inference	KServe‑compatible gRPC/REST for classification, detection, OCR, etc. Handles TensorFlow, ONNX, PaddlePaddle, OpenVINO IR, MediaPipe graphs, Python execution nodes, dynamic shapes.
Deployment	Docker images (CPU‑only, CPU+GPU/NPU), bare‑metal binaries for Linux/Windows, Helm charts for Kubernetes/OpenShift. Model repository can be local, S3, GCS, Azure Blob, or HuggingFace Hub.
Management	Model versioning, hot‑reload, Prometheus metrics, gRPC streaming, C API for embedding the server in native apps.
Hardware acceleration	Optimized for Intel Xeon CPUs, integrated/discrete GPUs, and Intel NPU via OpenVINO.

---

Typical use cases

• Serve a local LLM (e.g., Qwen3‑4B‑int4) behind an OpenAI‑compatible endpoint for chat applications, RAG pipelines, or internal tooling.
• Expose classic vision models (ResNet‑50, YOLO, OCR) to downstream services via KServe gRPC, enabling scalable inference in micro‑service architectures.
• Run multimodal agents that combine text, image and audio models, using the “AI Agents with MCP servers” demo.
• Deploy on edge or on‑premise servers where Intel CPUs/GPUs are available and cloud‑based APIs are undesirable.

---

Getting started (quick‑start)

1. Pull the Docker image

   docker pull openvino/model_server:latest   # CPU only
   # or
   docker pull openvino/model_server:latest-gpu   # CPU + Intel GPU/NPU
   

2. Run a model – example for a HuggingFace LLM (Qwen3‑4B‑int4) on Linux:

   mkdir -p $HOME/models
   docker run --rm -p 8000:8000 \
     --user $(id -u):$(id -g) -v $HOME/models:/models:rw \
     openvino/model_server:latest \
     --source_model OpenVINO/Qwen3-4B-int4-ov \
     --model_repository_path /models \
     --rest_port 8000
   

3. Call the API (Python, OpenAI client):

   from openai import OpenAI
   client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
   stream = client.chat.completions.create(
       model="OpenVINO/Qwen3-4B-int4-ov",
       messages=[{"role": "user", "content": "What are the 3 main tourist attractions in Paris?"}],
       stream=True)
   for chunk in stream:
       if chunk.choices[0].delta.content:
           print(chunk.choices[0].delta.content, end="")
   

4. Classic model example – serve a ResNet‑50 IR model via KServe gRPC:

   docker run --rm -d -u $(id -u) -v $(pwd):/models -p 9000:9000 \
     openvino/model_server:latest \
     --model_name resnet --model_path /models/resnet50.xml \
     --mean "[123.675,116.28,103.53]" --scale "[58.395,57.12,57.375]" \
     --layout "NHWC:NCHW" --port 9000
   

   Then use a Triton client (shown in the README) to send an image and read the top‑1 class.

---

Ecosystem & integration

• Client libraries – any HTTP/REST client works; official examples use the OpenAI Python SDK and Triton gRPC client.
• Model formats – TensorFlow, ONNX, PaddlePaddle, OpenVINO IR, MediaPipe, GGUF.
• Observability – Prometheus‑compatible metrics endpoint; logs follow standard Docker/Kubernetes conventions.
• Extensibility – C API for embedding the server, Python execution nodes for custom pre/post‑processing, and support for custom pipelines via MediaPipe graphs.

---

License & contributions

• License: Apache 2.0 (per the badge in the README).
• Contributing: Open to pull requests and issues on GitHub; security policy provided.

---

Bottom line: OpenVINO Model Server is a real, actively maintained inference serving platform that bridges the gap between Intel‑optimized hardware and modern AI workloads—from large language models to classic vision pipelines—while offering familiar OpenAI and KServe APIs for easy integration.