JingyunLiang/VRT

VRT – Video Restoration Transformer

What it is – VRT is a research‑grade PyTorch implementation of the Video Restoration Transformer described in the paper "VRT: A Video Restoration Transformer" (arXiv 2201.12288). It tackles a range of video‑restoration tasks such as video super‑resolution, deblurring, denoising, frame interpolation and space‑time super‑resolution.

Why it matters – Traditional video‑restoration networks either process frames one‑by‑one (recurrent) or use a short sliding window, limiting their ability to capture long‑range temporal information. VRT introduces a Temporal Mutual Self‑Attention (TMSA) mechanism that jointly estimates motion, aligns features and fuses information across many frames, while still allowing parallel processing. The authors report state‑of‑the‑art gains of up to 2.16 dB on nine benchmark datasets.

Key Features (as described in the README)

Multi‑task model – a single architecture works for video SR, deblurring, denoising, frame interpolation and space‑time SR.
Long‑range temporal modeling – TMSA splits the video into short clips for mutual attention, then shifts the sequence between layers to enable cross‑clip interaction.
Parallel warping – an additional warping branch further fuses neighboring‑frame information.
Pre‑trained weights – ready‑to‑download models for all supported tasks (released on the GitHub Releases page).
Benchmark‑level performance – improvements of +0.33 – 0.51 dB on video SR, +1.47 – 2.15 dB on deblurring, +1.56 – 2.16 dB on denoising, etc.
Colab demo – a one‑click Google‑Colab notebook lets you run the model without installing anything locally.

Quick Start (testing only)

# Clone the repo and install dependencies
git clone https://github.com/JingyunLiang/VRT
cd VRT
pip install -r requirements.txt   # Python 3.8+, PyTorch >=1.9.1, CUDA 11.1 recommended

# Example: video super‑resolution (6‑frame REDS model)
python main_test_vrt.py \
  --task 001_VRT_videosr_bi_REDS_6frames \
  --folder_lq testsets/REDS4/sharp_bicubic \
  --folder_gt testsets/REDS4/GT \
  --tile 40 128 128 \
  --tile_overlap 2 20 20

The script automatically downloads the required pretrained model and test data (except the Vimeo‑90K test set). Adjust --tile if you run out of GPU memory – smaller tiles use less memory but may slightly lower PSNR.

Training (high‑level overview)

Datasets – The README lists the exact training and testing splits for each task (e.g., REDS for video SR, Vimeo‑90K for SR/BI/BD, DVD/GoPro for deblurring, DAVIS for denoising, etc.).
LMDB conversion – For speed, the authors recommend converting PNG image sequences to LMDB format using the provided create_lmdb.py script from the KAIR toolbox.
No dataset preparation needed for testing – main_test_vrt.py will fetch the test sets automatically.

Resources

Paper: https://arxiv.org/abs/2201.12288
Supplementary PDF: https://github.com/JingyunLiang/VRT/releases/download/v0.0/VRT_supplementary.pdf
Pre‑trained models & visual results: https://github.com/JingdongLiang/VRT/releases
Colab demo: https://colab.research.google.com/gist/JingyunLiang/deb335792768ad9eb73854a8efca4fe0#file-vrt-demo-on-video-restoration-ipynb
Related work – The authors also point to their follow‑up Recurrent Video Restoration Transformer (RVRT) and earlier image‑restoration transformers (SwinIR, HCFlow, MANet, etc.).

Citation

If you use VRT in research, cite the original paper (arXiv 2201.12288). The README includes a ready‑to‑copy BibTeX entry under the Citation section.

License & Acknowledgements

The repository is released under the license specified in the LICENSE file (standard for academic code). It acknowledges the ETH Zurich Computer Vision Lab and Meta Inc. as the supporting institutions.

Bottom line – VRT is a fully‑featured, open‑source implementation of a cutting‑edge transformer for a broad suite of video‑restoration problems. It provides ready‑to‑run pretrained models, clear testing scripts, and documentation sufficient for researchers to reproduce the results or fine‑tune the model on their own video data.