AmusementClub/vs-mlrt

Efficient CPU/GPU ML Runtimes for VapourSynth (with built-in support for waifu2x, DPIR, RealESRGANv2/v3, Real-CUGAN, RIFE, SCUNet, ArtCNN and more!)

What it solves

It provides a set of high-performance AI inference runtimes for VapourSynth ML filters, allowing users to run AI-based video processing filters across a wide variety of hardware platforms including x86 CPUs and GPUs from Intel, NVIDIA, and AMD, as well as Apple SoCs.

How it works

The project bundles multiple specialized inference backends as VapourSynth plugins. It leverages industry-standard runtimes to optimize performance for specific hardware:

  • OpenVINO: Optimized for Intel CPUs and GPUs.
  • ONNX Runtime: Provides general CPU and CUDA GPU support, including CoreML for Apple SoC.
  • TensorRT/TensorRT-RTX: Highly optimized for NVIDIA GPUs using hardware-specific benchmarking.
  • MIGraphX: Optimized for AMD GPUs.
  • ncnn: Uses the Vulkan API to provide a lightweight, cross-vendor GPU runtime (NVIDIA, AMD, Intel).

To simplify the user experience, it includes a Python wrapper (vsmlrt.py) that provides a unified interface for selecting these different backends and managing bundled models.

Who it’s for

Video processing enthusiasts and developers using VapourSynth who want to apply AI filters to their video content with the best possible performance for their specific hardware.

Highlights

  • Broad Hardware Support: Covers almost all modern compute platforms (Intel, NVIDIA, AMD, Apple).
  • Optimized Performance: Includes specialized runtimes like TensorRT and MIGraphX for maximum GPU throughput.
  • Lightweight Alternative: Offers a Vulkan-based ncnn runtime for users who need a smaller installation footprint.
  • Unified Interface: A Python wrapper simplifies the process of switching between different inference backends.

Related

  • Project
    amd/RyzenAI-SW

    A suite of tools and runtime libraries for optimizing and deploying AI inference on AMD Ryzen AI-based PCs to maximize hardware acceleration.

  • Project
    nndeploy/nndeploy

    A high-performance AI deployment framework that uses visual workflows to deploy AI models across desktop, mobile, and edge devices.

  • Project
    lablup/mlxcel

    A high-performance native runtime and server for LLMs, VLMs, and audio models on Apple Silicon and NVIDIA CUDA, removing Python from the inference path.

  • Project
    NVIDIA/TensorRT-Edge-LLM

    A high-performance C++ inference runtime for deploying text, vision, audio, and speech models on NVIDIA Jetson, DRIVE, and DGX Spark edge platforms.

  • Project
    NVIDIA/TensorRT-LLM

    An optimization library for LLMs and visual generation models that uses specialized kernels and an efficient runtime to maximize inference throughput and minimize latency on NVIDIA GPUs.