amanchadha/iSeeBetter

What it solves

iSeeBetter addresses the problem of video super-resolution (VSR), where the goal is to upscale low-resolution videos to high-resolution versions. Unlike single-image super-resolution, which can lead to a lack of temporal coherency (flickering or inconsistency between frames), iSeeBetter aims to produce temporally consistent videos with high perceptual quality and fine-grained texture details.

How it works

The project uses a Generative Adversarial Network (GAN) architecture. The generator is a Recurrent Back-Projection Network (RBPN) that extracts both spatial information from the current frame (SISR) and temporal information from neighboring frames (MISR) using dense motion flow maps. This combined information is fused through a recurrent encoder-decoder mechanism and incorporated into the estimated frame via back-projection.

To ensure the output looks natural and avoids artifacts, it uses a discriminator from the SRGAN architecture. The model is trained using a four-fold loss function that combines:

• MSE Loss: Improves PSNR and SSIM metrics.
• Perceptual Loss: Captures fine details that MSE might miss.
• Adversarial Loss: Enhances the "naturality" of the image.
• Total-Variation (TV) Loss: Reduces noise and artifacts.

Who it’s for

Researchers and developers working on computer vision, video processing, and deep learning who want to implement or improve upon state-of-the-art video upscaling techniques.

Highlights

• Spatio-Temporal Approach: Utilizes both current and adjacent frames to recover missing details.
• GAN-Based: Uses an SRGAN discriminator to improve perceptual quality over traditional CNNs.
• SOTA Performance: Ranked #1 on the Video Super Resolution leaderboard on PapersWithCode for Vid4 4x upscaling at the time of publication.
• Comprehensive Loss Function: Combines four different loss types to optimize for both mathematical accuracy and visual realism.