AlibabaResearch/SparkDiffusion

SparkDiffusion: Accelerating DiT video generation by 265× via joint Sparsity, Distillation, Quantization, etc.

What it solves

SparkDiffusion is designed to accelerate the inference of Diffusion Transformer (DiT) video generation models, specifically targeting the Wan 2.1 and Wan 2.2 models. It addresses the high computational cost of dense multi-step generation, enabling up to 265x faster inference on a single GPU while maintaining generation quality.

How it works

The framework employs a three-pronged approach to achieve acceleration:

  1. Sparse Low-Rank Attention (RoLa): An efficient attention design that reduces the complexity of the attention mechanism during both training and inference.
  2. Few-Step Distillation (CrossDistill): A trajectory-level hybrid distillation technique that allows the model to generate high-quality video in significantly fewer steps (e.g., 3-4 steps).
  3. Custom High-Performance Operators: Self-developed, dependency-free inference operators that optimize the execution of the sparse attention mechanism.

Additionally, the framework supports weight-activation quantization and provides an end-to-end pipeline for sparse-attention finetuning and distillation.

Who it’s for

Developers and researchers working with video diffusion models who need to significantly reduce latency and computational requirements for text-to-video (T2V) and image-to-video (I2V) generation.

Highlights

  • Massive Speedups: achieves 200x+ end-to-end inference acceleration over dense baselines.
  • Wan Model Support: fully compatible with Wan 2.1 and Wan 2.2 architectures.
  • Integrated Pipeline: provides tools for sparse-attention finetuning, few-step distillation, and optimized inference.
  • Custom Operators: includes self-developed fused kernels for high-performance inference without external dependencies.
  • Pluggable Registry: allows custom attention variants to be integrated without modifying core code.

Related

  • Project
    TencentARC/WorldCrafter

    WorldCrafter is a research‑grade video generation model that creates camera‑controllable, consistent videos from a single image or a text prompt. It stores an implicit 3‑D‑aware memory so the scene stays coherent across viewpoints. The repo provides a full‑size Base model and a distilled Fast model, scripts for image‑to‑video and text‑to‑video inference, automatic prompt generation via Qwen‑VL, and an open‑source interactive web demo. Setup uses Python 3.11 with PyTorch 2.10 (CUDA 12.8) via uv or conda, and model weights are hosted on Hugging Face.

  • Project
    yizhi-chengzi/video-ai-talking

    A local web app that turns a silent face‑cam clip into a full talking‑head video using Alibaba Bailian VideoRetalk for lip‑sync, TTS from Bailian or Volcengine, optional DeepSeek script generation, and FFmpeg for final rendering.

  • Project
    vllm-project/vime

    Vime is an open‑source framework that combines Megatron‑LM training with vLLM inference to enable high‑performance reinforcement‑learning fine‑tuning of large language models. It offers a unified data‑flow loop, flexible custom generation hooks, and support for many modern LLMs (Qwen, DeepSeek, LLaMA 3).

  • Project
    Carasibana/ComfyUI-H3-FaceRefine

    ComfyUI‑H3‑FaceRefine adds custom nodes that detect, crop, re‑generate, and stitch back tiny faces in MiniMax H3 videos, fixing the model’s poor performance on small heads. It offers automatic and manual face‑selection workflows, robust tracking, optional cut detection, and integrates with existing MiniMax H3 pipelines.