NVlabs/LongLive

Long Video Gen Infrastructure

What it solves

LongLive 2.0 is an infrastructure designed to make long video generation more efficient and scalable. It addresses the high computational and memory costs associated with generating extended sequences of video frames, enabling real-time interactive generation and high-throughput inference.

How it works

The project implements a parallel infrastructure that leverages NVFP4 (a 4-bit floating point format) and sequence parallelism to optimize both training and inference.

For training, it supports autoregressive (AR) teacher-forcing training for text-to-video (T2V) and image-to-video (I2V) modalities, as well as few-step distillation (DMD distillation) to reduce the number of steps required for generation.

For inference, it utilizes NVFP4 weights and KV cache to reduce memory overhead, sequence parallel inference, and async decoding. It also incorporates techniques like multi-shot attention sinks to maintain stability over long durations.

Who it’s for

This infrastructure is intended for researchers and developers working on long-horizon video generation, specifically those using NVIDIA GPUs (SM90/H100) to maximize throughput and hardware-accelerated quantization.

Highlights

  • NVFP4 Support: Implements W4A4 quantization and NVFP4 KV cache for significantly faster inference (up to 45.7 FPS).
  • Causal Diffusion: Supports both AR training and DMD distillation for efficient, few-step video generation.
  • Parallelism: Features balanced sequence parallel training and sequence parallel inference to handle long sequences.
  • Versatile Modalities: Supports both text-to-video (T2V) and image-to-video (I2V) workflows.
  • FP8 PTQ: Includes TorchAO FP8 post-training quantization for BF16 checkpoints.

Related

  • Project
    ModelTC/LightX2V

    LightX2V is an open‑source inference framework for fast image and video generation (T2V, I2V, T2I, I2I, T2AV, etc.). It supports many state‑of‑the‑art models (MiniMax‑H3, Wan, HunyuanVideo, Qwen‑Image, etc.) and provides speed‑up techniques such as 4‑step distilled LoRAs, FP8/NVFP4 quantization, tensor/sequence parallelism, and disaggregated deployment on a wide range of hardware. Benchmarks show up to 3.9× faster inference than competing frameworks on H100 GPUs. The project offers Docker images, pip install, Gradio/ComfyUI front‑ends, extensive docs, and an online demo.

  • Project
    hao-ai-lab/FastVideo

    A unified post-training and real-time inference framework designed to accelerate video generation through sparse distillation and attention optimizations.

  • Dispatch
    vLLM-Omni Accelerates Inference with AutoRound Quantization

    vLLM-Omni now integrates Intel's AutoRound post-training quantization, enabling W4A16 quantization that cuts model size up to 62% while preserving accuracy and unlocking performance gains on Intel XPU and NVIDIA GPUs.

  • Project
    thu-ml/Causal-Forcing

    A framework for high-quality real-time interactive video generation using Causal ODE or Causal Consistency Distillation to enable efficient few-step autoregressive diffusion.

  • Project
    chenfengxu714/StreamDiffusionV2

    An interactive diffusion pipeline for real-time streaming video generation that optimizes throughput and latency across single and multi-GPU setups.