Tencent-Hunyuan/Prism

Prism – Dynamic Sparse Attention for 2K Joint Video‑Audio Generation

What it is – Prism is a research‑grade codebase that lets you train and run large‑scale video‑audio diffusion models at high resolutions (720 p, 1080 p, 2 K). The core novelty is a dynamic block‑sparse attention mechanism that adapts the attention pattern to local visual motion and audio‑visual coupling, cutting training time by ~2.5× while improving generation quality.

Why it matters – Full‑attention transformers scale quadratically with the number of tokens, which becomes prohibitive for high‑resolution video (spatial × temporal × audio). Prism’s attention partitions the token stream into macro‑zones and assigns each zone a custom block shape based on:

• Video feature variance (how quickly visual content changes)
• Audio‑to‑video cross‑attention norms (where sound is produced)

This yields a sparse pattern that keeps semantically coherent groups together, preserving important cross‑modal interactions without the massive compute cost of dense attention.

Key components

• Macro‑zone token layout – the video‑audio token sequence is split into spatio‑temporal blocks.
• Dynamic block sizing – each zone gets a block shape tuned to its content (finer blocks where motion or sound is strong).
• Hybrid block selection – per‑query sparsity is chosen on‑the‑fly, allowing the model to keep more tokens where needed and drop redundant ones elsewhere.
• Integration with MOVA – Prism builds on the MOVA video‑audio diffusion backbone, re‑using its pretrained VAE and diffusion pipelines.

What you can do with it

• Train native joint video‑audio diffusion models at 720 p, 1080 p, or 2 K using FSDP and sequence‑parallelism across multiple GPUs.
• Run inference on a single GPU (DeepSpeed) or multi‑GPU (FSDP) to generate videos from text prompts, audio prompts, or image‑to‑video seeds.
• Experiment with the block‑sparse attention knobs (ENABLE_BSA, BSA_SPARSITY, ENABLE_IVPQ_DYNAMIC_BLOCK) to trade speed vs. fidelity.

How to get started

1. Set up the environment – Python ≥ 3.10, CUDA ≥ 12.4, ffmpeg, plus PyTorch 2.7, FlashAttention, Diffusers, Transformers, etc.
2. Download the preview checkpoint from Hugging Face (FrancisRing/Prism).
3. Run the provided inference scripts (prism_infer.sh for single‑GPU/DeepSpeed, prism_infer_fsdp.sh for multi‑GPU FSDP). Adjust resolution, frame count, prompts, and sparsity settings in the command line.
4. Prepare your own training data – a CSV manifest describing latent paths, video/audio files, captions, and dimensions; then extract WAN‑VAE latents and bucket the data with the supplied build_dataset_csv.py script.
5. Launch training with the pdsh_train.sh wrapper, picking the appropriate resolution config (720 p, 1080 p, 2 K).

Resources – Project page, arXiv paper, Hugging Face model repo, YouTube/Bilibili demo videos, and a set of ready‑to‑run inference cases are linked at the top of the README.

---

TL;DR – Prism is a cutting‑edge framework for high‑resolution joint video‑audio diffusion, introducing a content‑aware dynamic sparse attention that speeds up training and improves output quality. The repo provides full training and inference pipelines, pretrained checkpoints, and detailed setup instructions.