sotayang/Awesome-Streaming-Video-Understanding

What it solves

Traditional video understanding models need the whole video before they can answer questions. This repository focuses on a harder, more practical problem: how to build AI assistants that process video as it happens, frame by frame, and decide when to speak, stay silent, or ask for more detail. It collects the best research so builders and researchers can find everything needed to create always-on, interactive video AI systems.

How it works

The list is organized around two core challenges in streaming video understanding. One challenge is deciding when to act: models use techniques like predicting a special token to stay quiet, using separate detectors, or checking uncertainty to trigger a response. The other challenge is how to keep processing efficiently over long, continuous streams: methods include managing the growing memory/cache, compressing history, using retrieval, and making computation sparser. The repository also groups key benchmarks and datasets used to evaluate these systems, and includes a comprehensive survey paper.

Who it’s for

Researchers and practitioners exploring the frontier of real-time, always-on video AI. It serves both as a reference for academic work and as a practical guide for developers building streaming video assistants, such as real-time narrators, fitness coaches, surveillance monitors, or embodied robots.

Highlights

• Curated list of 100+ papers with links, venues, and release dates, covering proactive and reactive streaming models
• Dedicated sections for benchmarks and datasets that evaluate multi-turn dialogue, real-time captioning, and proactive timing
• Includes a companion survey paper: "Towards Online Interactors: A Comprehensive Survey on Streaming Video Understanding"
• Organized by method family (token-based triggering, classification heads, memory management, etc.) for easy comparison
• Actively maintained with a welcoming contribution policy (PRs welcome)