microsoft/mlvc

What it solves

MLVC is a neural video codec designed for real-world deployment on consumer devices. It addresses the challenge of creating a learned video compression system that maintains high visual quality while achieving real-time performance (averaging 100 FPS) on commodity NPUs (Neural Processing Units) from Apple, Intel, and Qualcomm.

How it works

MLVC uses a learned approach to video compression, providing a full pipeline for training and deploying models. It includes:

• Training Pipeline: Supports training for both image (I-frame) and video (P-frame) models, with options for PSNR-based or perceptual-based optimization.
• Hardware Optimization: Includes conversion tools to export models to CoreML (Apple), ONNX for OpenVINO (Intel), and QNN (Qualcomm) to leverage hardware acceleration.
• Entropy Coding: A production-ready C++ entropy coder optimized for real-time deployment to handle the actual bitstream production.
• Evaluation Suite: Tools to compute BD-rate, RD curves, and various quality metrics (PSNR, MS-SSIM, LPIPS, etc.) to compare performance against traditional codecs like H.265/HEVC.

Who it’s for

Researchers and developers working on learned video compression, neural codecs, and high-performance AI-driven media processing on edge devices.

Highlights

• Real-time Performance: Achieves approximately 100 FPS for encoding and decoding on commodity NPUs.
• Hardware Robustness: Native support for Apple, Intel, and Qualcomm NPUs via specialized export tools.
• Significant Efficiency Gains: Over 70% MOS-based BD-rate improvement over hardware HEVC.
• Comprehensive Tooling: Provides pre-trained checkpoints, training code, and benchmarking tools.