jamichss/Stream-DiffVSR

Stream‑DiffVSR – Low‑latency, streamable video super‑resolution

What it does – Stream‑DiffVSR is a diffusion‑based video‑super‑resolution (VSR) system that works online: it only looks at past frames, so it can be used on live video streams. The model runs a distilled 4‑step denoiser, an Auto‑Regressive Temporal Guidance (ARTG) block that aligns motion cues, and a lightweight temporal decoder with a Temporal Processor Module (TPM) to keep details and temporal coherence.

Why it matters – Traditional diffusion VSR needs future frames and many denoising steps, giving latencies of several seconds to minutes. Stream‑DiffVSR processes a 720p frame in ~0.33 s on an RTX 4090, a >130× speed‑up over the previous online state‑of‑the‑art (TMP). This makes diffusion‑quality VSR feasible for real‑time or low‑latency applications such as video calls, streaming, or live broadcasting.

Key components

• Four‑step distilled denoiser – reduces the usual 20‑30 diffusion steps to just four, cutting inference time.
• Auto‑Regressive Temporal Guidance (ARTG) – injects motion‑aligned information from earlier frames while denoising the current latent.
• Temporal Processor Module (TPM) – part of a temporal‑aware decoder that refines spatial detail and enforces consistency across frames.

Getting started

1. Clone & set up – git clone https://github.com/jamichss/Stream-DiffVSR.git then create the conda environment from requirements.yml (or the RTX‑specific requirements-cu12.txt).
2. Run inference – the CLI inference.py pulls the pretrained model from Hugging Face automatically:

   python inference.py \
       --model_id Jamichsu/Stream-DiffVSR \
       --in_path  /path/to/low‑res/frames \
       --out_path /path/to/high‑res/output \
       --num_inference_steps 4
   

   Input should be a folder of sub‑directories, each containing an ordered list of PNG frames.
3. Optional speed boost – add --enable_tensorrt (requires specifying output height/width) to compile a TensorRT engine. This trades a tiny quality loss for higher throughput.
4. Gradio UI – install requirements-app.txt and launch python app.py to get a web UI that extracts frames from a video, runs the model, and re‑assembles the upscaled video with original audio.
5. Docker – a ready‑to‑run image is provided; build with docker build -t stream-diffvsr-gradio . and run with GPU access.

Training – The repo also contains scripts for reproducing the authors’ training pipeline:

• Stage 1: UNet training with rollout distillation.
• Stage 2: Train the TPM inside the temporal decoder.
• Stage 3: Train the ARTG while freezing the UNet and TPM.

Training uses the REDS and Vimeo‑90K datasets (paths set in dataset/config_*.yaml). The UNet is initialized from StableVSR weights, and the temporal decoder weights come from TAESD‑X4, both automatically downloaded.

Where to find the model – Pre‑trained weights are hosted on Hugging Face under Jamichsu/Stream-DiffVSR. The inference script fetches them on‑the‑fly, so no manual download is needed.

Citation – If you use the code or model, cite the arXiv paper (arXiv:2512.23709).

---

All information above is taken directly from the repository’s README.