openvinotoolkit/model_server
A scalable inference server for models optimized with OpenVINO™
What it solves
It simplifies the deployment of AI models by hosting them on a server and making them accessible via standard network protocols. This decouples the client application from the model framework, hardware, and infrastructure, allowing for lightweight clients and easier updates without exposing model weights directly to the end user.
How it works
Implemented in C++ and optimized for Intel architectures, the system acts as an inference server that receives requests via gRPC or REST APIs. It can pull models from local storage, object storage, or the HuggingFace Hub and execute them using OpenVINO. It supports various APIs including OpenAI, Cohere, KServe, and TensorFlow Serving, and can be deployed on Bare Metal, Docker, or Kubernetes.
Who it’s for
Developers and DevOps engineers building microservices-based AI applications who need a high-performance, scalable way to serve models across edge or cloud environments.
Highlights
- Broad API Compatibility: Supports OpenAI, Cohere, KServe, and TensorFlow Serving APIs for text, image, and speech generation.
- Flexible Deployment: Runs on Windows, Linux, Docker, and Kubernetes.
- Model Management: Includes features like model versioning and runtime updates.
- Advanced Scheduling: Features a Directed Acyclic Graph (DAG) Scheduler for complex pipelines.
- Multi-Framework Support: Compatible with TensorFlow, PaddlePaddle, and ONNX models.