chenfengxu714/StreamDiffusionV2

StreamDiffusionV2 – Real‑time Video Generation with Diffusion

What it is – StreamDiffusionV2 is an open‑source library that lets you run diffusion‑based video generation as a streaming process. Instead of waiting for a whole video to be generated, the system processes short video chunks on‑the‑fly, keeping a rolling KV cache so that each new chunk can reuse the work already done. It supports single‑GPU, multi‑GPU pipeline‑parallel, and a lightweight web UI for interactive use.

Why it matters – Traditional text‑to‑video diffusion models are batch‑oriented and slow (minutes per second of video). StreamDiffusionV2 achieves high frame‑rates (real‑time or near‑real‑time) by:

• Splitting the video into overlapping chunks and re‑using latent states via a ring‑buffer KV cache.
• Allowing flexible numbers of denoising steps per chunk (trade‑off speed vs. quality).
• Providing optional acceleration paths: a fast TAEHV‑VAE decoder, Flash‑Attention, and TensorRT support.
• Scaling from a single GPU to multi‑GPU pipeline parallelism, with a “schedule‑block” option for better throughput on larger rigs.

Key components

Component	Role
StreamDiffusionV2Pipeline	High‑level Python API that handles chunking, encoding, denoising, and decoding.
run_v2v.sh	Unified CLI entry‑point for offline inference (single, single‑wo, pipe).
demo/	Minimal FastAPI/Gradio web UI for interactive generation.
streamv2v/	Example scripts showing how to drive the pipeline programmatically.
KV‑cache implementation	Ring‑buffer that stores attention keys/values for the causal DiT backbone, enabling sliding‑window attention.

Installation – The package is published on PyPI and can be installed with optional Flash‑Attention support:

conda create -n streamdiffusionv2 python=3.10 -y
conda activate streamdiffusionv2
pip install streamdiffusionv2            # core
pip install "streamdiffusionv2[flash-attn]"  # faster attention (requires CUDA)
# For the newest NVIDIA Blackwell GPUs
pip install torch==2.11.0 torchvision==0.26.0

If you prefer a local checkout, pip install . works the same way.

Getting the model – The library ships inference code only; you need to download a checkpoint:

# 1.3 B model (default for demo)
huggingface-cli download --resume-download Wan-AI/Wan2.1-T2V-1.3B --local-dir wan_models/Wan2.1-T2V-1.3B
huggingface-cli download --resume-download jerryfeng/StreamDiffusionV2 --local-dir ./ckpts --include "wan_causal_dmd_v2v/*"

# 14 B model (higher quality)
huggingface-cli download --resume-download Wan-AI/Wan2.1-T2V-14B --local-dir wan_models/Wan2.1-T2V-14B
huggingface-cli download --resume-download jerryfeng/StreamDiffusionV2 --local-dir ./ckpts --include "wan_causal_dmd_v2v_14b/*"

Optionally pull the lightweight TAEHV‑VAE decoder:

curl -L https://github.com/madebyollin/taehv/raw/main/taew2_1.pth -o ckpts/taew2_1.pth

Quick Python example (single‑GPU)

import numpy as np
from streamdiffusionv2 import (
    StreamDiffusionV2Pipeline,
    load_video,
    export_video,
)

pipe = StreamDiffusionV2Pipeline(
    checkpoint_folder="ckpts/wan_causal_dmd_v2v",
    mode="single",
)
pipe.prepare("A dog walks on the grass, realistic")

video = load_video("examples/original.mp4", height=480, width=832)
chunks = []
noise = pipe.noise_scale

for chunk in pipe.chunk_video(video):
    enc = pipe.encode_chunk(video, chunk, previous_noise_scale=noise,
                           initial_noise_scale=pipe.noise_scale)
    noise = enc.noise_scale
    den = pipe.denoise_chunk(enc)
    if den is None:
        continue
    chunks.append(pipe.decode_chunk(den))

out = np.concatenate(chunks, axis=0)
export_video(out, "outputs/python_single.mp4", fps=16)

The same pipeline can be run in pipeline mode for multi‑GPU inference with a single call:

from streamdiffusionv2 import run_video_to_video
run_video_to_video(
    mode="pipe",
    checkpoint_folder="ckpts/wan_causal_dmd_v2v",
    video_path="examples/original.mp4",
    prompt="A dog walks on the grass, realistic",
    output_path="outputs/python_pipe.mp4",
    gpu_ids=[0,1],
    num_gpus=2,
)

CLI usage – The run_v2v.sh script abstracts the same steps:

# single‑GPU streaming
./run_v2v.sh single
# single‑GPU without stream‑batch (slightly different memory pattern)
./run_v2v.sh single-wo
# multi‑GPU pipeline
./run_v2v.sh pipe
# add profiling
./run_v2v.sh pipe --profile
# enable fast TAEHV decoder
./run_v2v.sh single --use_taehv

Common flags (--config_path, --checkpoint_folder, --video_path, --prompt_file_path, --height, --width, --fps, --step, --use_taehv) can also be supplied via environment variables.

Web demo – After installing Node.js 18, start the UI with:

cd demo
USE_TAEHV=1 python -m streamdiffusionv2.web_demo   # or the provided launch script

Then open http://localhost:7860 in a browser.

Training – The repo contains a separate user-study branch that builds on the CausVid codebase. It provides scripts for training the causal DiT backbone used at inference time.

Roadmap – The authors list upcoming work such as a dynamic scheduler, FP8 quantisation, and deeper TensorRT integration.

Citation – If you use the library in research, cite the MLSys 2026 paper (BibTeX provided in the README).

---

StreamDiffusionV2 is a production‑ready toolkit for anyone who wants to experiment with or deploy real‑time diffusion video generation, offering both a simple Python API and a scalable multi‑GPU backend.