microsoft/mlvc

MLVC: Multi-platform Learned Video Codec for Real-World Deployment

TL;DR

MLVC is a neural video codec that runs in real‑time on consumer‑grade NPUs (Apple, Intel, Qualcomm). It beats hardware HEVC by >70 % BD‑rate while encoding/decoding at ~100 fps for 540p video. The repo ships pretrained models, a full training pipeline, tools to convert the models to CoreML/ONNX/QNN, a C++ entropy coder, and benchmarking utilities. A companion C++ library (libmlvc) provides production‑ready encoding/decoding.

What it does

  • Neural video compression – learns I‑frames (image) and P‑frames (motion‑compensated video) with either PSNR‑or perceptual‑oriented objectives.
  • Real‑time on‑device inference – targets the NPUs found in phones, tablets and laptops; the reference implementation reaches ~100 fps at 540p.
  • Hardware‑agnostic deployment – export pipelines produce CoreML (Apple), ONNX for OpenVINO (Intel) and QNN (Qualcomm), plus a C++ entropy coder for actual bit‑stream generation.
  • Benchmarking suite – computes BD‑rate, RD curves and a range of quality metrics (PSNR, MS‑SSIM, LPIPS, VIF, DeQA) against HEVC anchors.

Who might use it

Audience Why it matters
Research labs A ready‑to‑run codebase for training and evaluating learned video codecs, with reproducible datasets and anchors.
Mobile/embedded developers Exported CoreML/ONNX/QNN models let you integrate a state‑of‑the‑art codec into apps without building a custom ML pipeline.
Product teams The companion libmlvc C++ library offers a drop‑in encoder/decoder that can run on existing NPU‑accelerated runtimes.
Benchmarking enthusiasts Built‑in tools for BD‑rate, RD‑curve and perceptual metrics make it easy to compare against HEVC or other learned codecs.

How to get started (quick‑start)

  1. Install the UV package manager (recommended) and Python 3.12+. ```bash uv sync --extra onnxruntime # pick the backend you need
2. **Install the entropy coder** (C++ compiler required).  ```bash
uv pip install packages/msrtc_rans
  1. Download a pretrained checkpoint (e.g., mlvc-psnr-v1.ckpt).
  2. Run the demo notebook (video/notebooks/demo.ipynb) or the one‑liner:

uv run convert.py export --model-version dmc61sbr_reglu

   This exports the default PSNR model to the format matching your platform.
5. **Encode/decode** using the C++ library (`libmlvc`) or the Python wrappers that call the exported model and the RANS entropy coder.

## Technical details (from the README)
- **Model size**: 18.3 M parameters for the full‑scale MLVC, 5.4 M for the lightweight MLVC‑S.
- **Training data**: OpenVidHD (1 M HD clips) and VCD (various resolutions).  Anchors are provided for HEVC hardware encoders and the DCVC‑RT reference.
- **Losses**: PSNR‑oriented (MSE) and perceptual (LPIPS, DeQA).  Optional face‑region metrics via RetinaFace/FarL.
- **Conversion workflow**: `convert.py` handles export, validation, profiling and bundling.  Input dimensions must be multiples of 16.
- **Supported runtimes**: CoreML (Apple), ONNX Runtime (CPU, GPU, OpenVINO, Windows ML), QNN (Qualcomm).
- **Benchmarking**: `compute_benchmark_metrics.py` and `calculate_frame_sequence_metrics.py` generate BD‑rate, bpp, and advanced perceptual scores.
- **Citation**: The authors ask to cite the arXiv paper (2606.28027, 2026).

---

## Limitations & open questions
- **Resolution focus**: The real‑time claim is demonstrated at 540p; higher resolutions will need more powerful NPUs or model scaling.
- **Hardware dependence**: Performance hinges on the presence of a supported NPU and the C++ entropy coder compiled for the target platform.
- **Perceptual quality**: The perceptual models rely on external metrics (LPIPS, DeQA) and extra pretrained weights, adding complexity to the pipeline.
- **Dataset licensing**: Users must obtain OpenVidHD and VCD datasets separately; the repo only provides scripts to prepare them.

---

## Quick reference (commands)
```bash
# Install dependencies (choose backend)
uv sync --extra onnxruntime          # CPU
uv sync --extra onnxruntime-gpu      # NVIDIA GPU
uv sync --extra onnxruntime-qnn      # Qualcomm NPU
uv sync --extra onnxruntime-openvino # Intel NPU

# Export a model (default CoreML for Apple)
uv run convert.py export --model-version dmc61sbr_reglu

# Export to ONNX for Intel
uv run convert.py export --model-version dmc61sbr_reglu \
    --model-type onnx --target-device intel

# Run a validation test (BD‑rate against HEVC anchor)
uv run convert.py test --model-path ./output/models/.../coreml-apple/960x544 run_validation_test

# Train an I‑frame model
uv run train_image.py --config configs/train_image-dcvcrt.yaml \
    --data_mount ~/mlvc --checkpoints_mount ~/mlvc/job-outputs --save_dir checkpoints/dmci

# Train a P‑frame model
uv run train_video.py --config configs/train_video-mlvc.yaml \
    --data_mount ~/mlvc --checkpoints_mount ~/mlvc/job-outputs --save_dir checkpoints/mlvc

Where to find more

Related

  • Project
    microsoft/DCVC

    An ultra-fast neural video codec that uses a chunk-based coding framework to achieve high compression efficiency and rapid encoding/decoding speeds.

  • Project
    krrishnarraj/clpeak

    clpeak is an open‑source cross‑platform benchmark that measures peak compute throughput, memory bandwidth and latency on CPUs, GPUs and NPUs. It supports many backends (CUDA, ROCm, Metal, oneAPI, Vulkan, OpenCL, ONNX Runtime, Core ML, LiteRT), offers both CLI and GUI, and can export results as JSON for comparison.

  • Project
    openvinotoolkit/openvino

    An open-source toolkit for optimizing and deploying deep learning models across CPU, GPU, and NPU hardware to boost inference performance.

  • Project
    Blaizzy/mlx-vlm

    A package for inference and and fine-tuning of Vision Language Models and Omni Models on Mac using MLX, supporting multimodal inputs and speculative decoding for faster generation.

  • Project
    OpenBMB/MiniCPM-V

    Open‑source multimodal LLMs (MiniCPM‑V 4.7 and MiniCPM‑o 4.5) that run on‑device, support vision‑language (images, video, docs) and full‑duplex audio‑visual interaction, with MoE inference optimised for phones.