google-deepmind/videoprism

VideoPrism – A General‑Purpose Video Encoder

What it is – VideoPrism is a foundation‑style visual encoder for video. It is a JAX/Flax model that can turn raw video clips into rich spatio‑temporal embeddings, and it also ships a paired video‑text encoder for cross‑modal retrieval and related tasks.

Why it matters – The authors pre‑trained the model on an enormous hybrid dataset (1 B image‑text pairs, 36 M high‑quality video‑text pairs, and 582 M noisy video clips). With a single frozen backbone they achieve state‑of‑the‑art results on 31 out of 33 public video‑understanding benchmarks, covering classification, retrieval, localization, captioning and QA.

How to use it – After installing the package (pip install .), you can load a checkpoint with the helper videoprism.models.get_model and load_pretrained_weights. The README provides two short code snippets:

import jax
from videoprism import models as vp

# Video‑only encoder
model = vp.get_model('videoprism_public_v1_base')
state = vp.load_pretrained_weights('videoprism_public_v1_base')

@jax.jit
def encode(video):
    return model.apply(state, video, train=False)

For the video‑text version you also load a SentencePiece tokenizer (c4_en) and call the model with both video tensors and tokenised text.

Ready‑made demos – The repo ships three Colab notebooks that walk you through:

• Video encoder demo – raw video → token embeddings.
• Video‑text encoder demo – compute joint video‑text embeddings for zero‑shot retrieval.
• Video classification demo – fine‑tune a lightweight head on top of the frozen backbone.

Models released – Four checkpoint families are available on Hugging Face:

Variant	Size	Params	Typical input	Checkpoint
VideoPrism‑B (base)	458 MB	114 M	16‑frame, 288×288 clips	videoprism-base-f16r288
VideoPrism‑L (large)	1.42 GB	354 M	8‑frame, 288×288 clips	videoprism-large-f8r288
VideoPrism‑LvT‑B (video‑text base)	991 MB	248 M	–	videoprism-lvt-base-f16r288
VideoPrism‑LvT‑L (video‑text large)	2.30 GB	580 M	–	videoprism-lvt-large-f8r288

Performance highlights (frozen backbones) –

• VideoGLUE (classification, action detection, etc.) – VideoPrism‑L reaches 85.0 % on Kinetics‑400 and 34.5 % on AVA‑K, beating prior SOTA.
• Zero‑shot video‑text retrieval – VideoPrism‑LvT‑L hits 75.0 % recall@1 on VATEX (video‑to‑text) and 57 % on VATEX (text‑to‑video).
• Zero‑shot video classification – VideoPrism‑LvT‑L scores 72.4 % on K400, surpassing earlier baselines.

Roadmap – The only listed TODO is adding PyTorch support, which would broaden accessibility beyond the JAX ecosystem.

License & disclaimer – Code is Apache 2.0; model weights and other assets are CC‑BY. The repository notes that this is not an official Google product.

---

Bottom line – VideoPrism gives researchers a ready‑to‑use, high‑capacity video encoder (and a video‑text variant) that can be plugged into downstream tasks with minimal fine‑tuning, and it comes with clear Colab notebooks and pre‑trained checkpoints.