openvinotoolkit/openvino
OpenVINO™ is an open source toolkit for optimizing and deploying AI inference
What it solves
OpenVINO is an open-source toolkit designed to optimize and deploy deep learning models across a wide variety of hardware platforms. It solves the problem of inefficient model deployment by boosting performance for tasks like computer vision, automatic speech recognition, generative AI, and natural language processing.
How it works
OpenVINO allows users to convert models trained in popular frameworks—such as PyTorch, TensorFlow, ONNX, Keras, PaddlePaddle, and JAX/Flax—into an optimized format. Once converted, the model is compiled for specific hardware, such as CPUs (x86, ARM), GPUs (Intel integrated and discrete), and AI accelerators (NPUs), allowing for efficient inference without needing the original training frameworks.
Who it’s for
It is intended for developers and AI engineers who need to deploy deep learning models on edge devices or cloud environments, specifically those targeting Intel-based hardware and other compatible platforms.
Highlights
- Broad Framework Support: Compatible with PyTorch, TensorFlow, ONNX, and Hugging Face Hub models via Optimum Intel.
- Hardware Flexibility: Supports inference on CPU, GPU, and NPU.
- GenAI Capabilities: Includes a dedicated GenAI API for optimized pipelines and performance for LLMs.
- Extensive Ecosystem: Integrates with tools like NNCF for quantization and the OpenVINO Model Server for scalable serving.