fudan-generative-vision/Hallo-Live

What it solves

Hallo-Live addresses the challenge of generating synchronized audio and video for digital avatars in real-time. Traditional methods often struggle with high latency or poor synchronization between speech and lip movements; this framework enables streaming generation of talking avatars from text prompts with low latency and high visual fidelity.

How it works

The system uses a causal dual-stream Diffusion Transformer (DiT) model to generate audio and video simultaneously. It operates in two main stages:

1. ODE Initialization: A pretrained dual-stream DiT is adapted to a streaming setting using a cross-modal future-expanding block-causal mask.
2. Self-Rollout & DMD: The model performs autoregressive self-rollout using audio-video KV caches and optimizes the generated trajectory using reward-weighted dual-stream Distribution Matching Distillation (DMD).

To further reduce latency, the framework supports parallel VAE decoding, where the diffusion model and VAE decoders run on separate GPUs to overlap generation and decoding.

Who it’s for

This project is designed for researchers and developers building real-time interactive AI avatars, digital humans, or streaming content generation tools that require tight lip-sync accuracy and high-quality speech.

Highlights

• Real-time Performance: Achieves 20.38 FPS with only 0.94 seconds of latency on two NVIDIA H200 GPUs.
• Joint Generation: Simultaneously produces both synchronized video and speech from a single text prompt.
• Versatile Styles: Supports various character types, including realistic humans, 3D anime, and hand-drawn anime styles.
• Low-Latency Architecture: Utilizes a causal DiT and KV caching to enable streaming inference.