vipshop/cache-dit

What it solves

Cache-DiT is a PyTorch-native inference engine designed to accelerate the execution of Diffusion Transformers (DiTs). It addresses the high computational cost and memory requirements of large-scale DiT models used in image, video, and audio generation, enabling faster inference speeds and more efficient hardware utilization.

How it works

Built on top of the Hugging Face Diffusers library, Cache-DiT employs several optimization techniques to reduce latency and memory footprint:

• Hybrid Cache Acceleration: Implements various caching strategies (such as DBCache, TaylorSeer, SCM, and DMD) to avoid redundant computations.
• Parallelism: Supports a wide array of parallelism strategies, including Context Parallelism, Tensor Parallelism, and hybrid 2D/3D parallelism, with dedicated support for Text Encoders, VAEs, and ControlNets.
• Memory Management: Provides quantization (W8A8, W4A4) and bucket-style layerwise CPU offloading to minimize GPU memory usage.
• Hardware Compatibility: Runs natively on NVIDIA GPUs, Ascend NPUs, and AMD GPUs, and is compatible with PyTorch compilation.

Who it’s for

This engine is intended for developers and researchers working with DiT-based generative AI models (such as FLUX, CogVideoX, and HunyuanVideo) who need to optimize their inference pipelines for production or high-performance computing environments.

Highlights

• Broad Model Support: Supports over 40 DiT pipeline families and 120+ variants from Diffusers.
• Huge Speedups: Capable of achieving up to 9x speedup through the combination of caching, context parallelism, and compilation.
• Deep Integration: Fully integrates with popular frameworks like SGLang Diffusion, vLLM-Omni, TensorRT-LLM, and ComfyUI.
• Agentic Integration: Includes a model-integration SKILL to help users integrate new DiT pipelines using coding agents like GitHub Copilot or Claude Code.