OpenVDN/vdn-minimax-h3

What is VDN‑Minimax‑H3?

VDN‑H3 (Video DeltaNet on MiniMax H3) is a hybrid‑attention video diffusion model that builds on the MiniMax H3 backbone. It adds a lightweight linear‑attention branch and a few LoRA adapters, letting the model generate 14‑second, 768p videos faster than real‑time while keeping visual quality close to the original dense model.

---

Key ideas

Idea	Why it matters
Hybrid architecture	One branch uses cheap linear attention for speed; the other keeps a full softmax attention branch for quality and temporal consistency.
Plug‑and‑play LoRA adapters	The extra linear branch and two tiny LoRA modules are merged at inference time, leaving the original MiniMax weights untouched.
FP8 inference + FlashAttention‑4	Mixed‑precision (FP8) and the newest FlashAttention kernels give up to 3×‑5× speed‑ups on modern NVIDIA H200/B200 GPUs.
Open‑source stack	Both the optimized inference code and the full training recipes are released, so anyone can reproduce or fine‑tune the model.

---

Getting started (quick‑start)

1. Clone & create a conda env (Python 3.12, PyTorch 2.13+CUDA 12.9). Install the uv package manager and run uv pip install -e . to pull in all dependencies, including the pre‑release flash‑attn‑4.
2. Patch Diffusers – a helper script (scripts/setup_diffusers.sh) applies the required changes to the HuggingFace Diffusers library.
3. Download the weights (≈ 82 GB) with the HuggingFace CLI:

   hf download OpenVDN/vdn-minimax-h3 --local-dir ckpts
   

   The directory will contain the MiniMax‑H3 base model (≈ 72 GB) and two VDN‑H3 checkpoints (≈ 4‑5 GB each).
4. Run a single‑GPU test:

   bash scripts/inference/8nfe_tuned_fp8.sh
   

   The script compiles the Flash kernels on first run and then denoises an 8‑step video in ~11 s on an 8‑GPU B200 node (or ~6 s on a single B200).
5. Render your own prompt – encode a text prompt with the Qwen3‑VL‑32B VLM, then call the inference script:

   python src/inference/encode_prompt.py --prompt "..." --out prompts/mine.pt
   python src/inference/infer.py \
     --config configs/inference/8nfe_tuned_fp8.yaml \
     checkpoint=ckpts/stage-dmd-step-250 \
     render.prompt_file=prompts/mine.pt \
     render.out=results/mine.mp4
   

   (The README recommends rewriting prompts with MiniMax‑H3‑Context‑IR for best quality.)

---

Performance numbers (steady‑state denoising, 768p, 14.4 s video)

GPU type	#GPUs	Seconds per NFE	50‑step VDN‑H3 (≈ 9 min)	8‑step VDN‑H3
H200	1	11.2	9.4 min	90.5 s
H200	8	2.29	1.9 min	18.3 s
B200	1	6.41	5.3 min	51 s
B200	8	1.40	1.2 min	11.23 s

(These timings exclude model loading, warm‑up, VAE decoding, and MP4 encoding.)

---

Training pipeline

VDN‑H3 is trained in four stages on top of a frozen MiniMax‑H3 transformer:

Stage	What is trained	Steps
A1	Linear‑attention branch (layer‑wise alignment)	200
A2	Same branch, end‑to‑end fine‑tune	500
B	LoRA adapters on QKV/O projections + linear branch	2000
DMD	8‑step “turbo” LoRA (data‑free, initialized from a community LoRA)	250

Training scripts live under src/training/ and scripts/training/. Data must be pre‑encoded latents (video, audio, text) following the MiniMax‑H3 format; a video_index.jsonl points to each clip’s video latent file.

---

License & attribution

• Code – Apache 2.0 (see LICENSE).
• Model weights – Distributed separately on HuggingFace under the MiniMax H3 Community License (not covered by Apache 2.0).
• Cite the work with the provided BibTeX entry.

---

TL;DR

VDN‑Minimax‑H3 is an open‑source, hybrid‑attention video diffusion model that adds a fast linear‑attention branch and tiny LoRA adapters to MiniMax H3, achieving near‑real‑time generation of high‑resolution videos. The repo supplies everything needed to run inference on a single GPU or an 8‑GPU cluster, and includes full training recipes for reproducing the model.