NVlabs/AutoGaze

What it solves

Video understanding models (ViTs and multimodal LLMs) become extremely expensive when processing high-resolution, high-frame-rate, or long videos because they must attend to every patch in every frame. AutoGaze addresses this by automatically selecting the informative patches and dropping redundant ones, so downstream models can process far fewer tokens without losing information. This makes it feasible to scale video models to 4K resolution and 1K-frame videos.

How it works

AutoGaze is an autoregressive "gazing" model: given a video, it predicts a sequence of gaze positions that point to informative patches, one patch at a time, and uses those to construct a reduced set of tokens. The model is trained with reinforcement learning (GRPO or next-token prediction) using a task objective such as VideoMAE reconstruction, where the reward measures how well the selected patches preserve the video's information. The README describes a modular codebase with components for gaze models, tasks (e.g., video MAE reconstruction), RL algorithms, datasets, and vision encoders (SigLIP is already adapted). It also demonstrates integration by scaling NVILA-8B to a "HD-Video" variant that uses AutoGaze to prune patches before its SigLIP encoder and LLM.

Who it's for

Researchers and engineers working on efficient video understanding with ViTs or multimodal LLMs. It is especially useful for anyone building systems that need to handle long, high-resolution videos under compute or memory constraints, and who want a ready-made, trainable patch-selection module plus integration examples.

Highlights

• Selects informative patches autoregressively and removes redundant ones, cutting token counts for downstream models.
• Scales ViTs/MLLMs to 4K resolution and 1K-frame videos without information loss.
• Released as a CVPR 2026 highlight paper (accepted CVPR 2026).
• Open-sources the pre-trained AutoGaze model, training data, and a new HLVid benchmark for high-resolution long-form video QA.
• Provides modular code (models, tasks, RL algorithms, vision encoders) so new tasks and models can be added easily.
• Includes an integration example with NVILA-8B-HD-Video and guidance for adapting SigLIP.