fudan-generative-vision/Hallo-Live

[ACM MM 2026] Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation

What it solves

Hallo-Live addresses the challenge of generating synchronized audio and video for digital avatars in real-time. Traditional methods often struggle with high latency or poor synchronization between speech and lip movements; this framework enables streaming generation of talking avatars from text prompts with low latency and high visual fidelity.

How it works

The system uses a causal dual-stream Diffusion Transformer (DiT) model to generate audio and video simultaneously. It operates in two main stages:

  1. ODE Initialization: A pretrained dual-stream DiT is adapted to a streaming setting using a cross-modal future-expanding block-causal mask.
  2. Self-Rollout & DMD: The model performs autoregressive self-rollout using audio-video KV caches and optimizes the generated trajectory using reward-weighted dual-stream Distribution Matching Distillation (DMD).

To further reduce latency, the framework supports parallel VAE decoding, where the diffusion model and VAE decoders run on separate GPUs to overlap generation and decoding.

Who it’s for

This project is designed for researchers and developers building real-time interactive AI avatars, digital humans, or streaming content generation tools that require tight lip-sync accuracy and high-quality speech.

Highlights

  • Real-time Performance: Achieves 20.38 FPS with only 0.94 seconds of latency on two NVIDIA H200 GPUs.
  • Joint Generation: Simultaneously produces both synchronized video and speech from a single text prompt.
  • Versatile Styles: Supports various character types, including realistic humans, 3D anime, and hand-drawn anime styles.
  • Low-Latency Architecture: Utilizes a causal DiT and KV caching to enable streaming inference.

Related

  • Project
    Alibaba-Quark/LiveAvatar

    Live Avatar is a research‑grade system that turns spoken audio (plus an optional reference image or text prompt) into a continuous, real‑time video of a talking avatar. Built on a 14 B diffusion backbone (WanS2V‑14B) with a LoRA fine‑tune, the authors achieve ~45 FPS streaming on multi‑GPU H800 hardware and can generate videos longer than 10 000 seconds via block‑wise autoregressive processing. The repo provides Conda‑based installation, multi‑GPU and single‑GPU inference scripts, optional FP8 quantisation and JIT compilation for speed, and a Gradio UI. Model weights are hosted on Hugging Face under Apache 2.0.

  • Project
    lipku/LiveTalking

    A real-time interactive streaming engine for digital humans that synchronizes audio and video for lifelike AI-driven dialogues.

  • Project
    PunithVT/ai-avatar-system

    AvatarAI is an open‑source, production‑grade system that turns any portrait photo and a short voice sample into a real‑time, lip‑synced talking avatar. It streams Whisper‑transcribed speech to a selectable LLM (Claude, GPT‑4o, or local Ollama/vLLM), feeds the generated text to a multilingual TTS (Chatterbox/XTTS), and renders video frames with MuseTalk V1.5. The stack includes Next.js front‑end, FastAPI back‑end, PostgreSQL, Redis, Celery, Prometheus, and Docker‑Compose, with optional one‑click AWS GPU deployment. Features include zero‑shot voice cloning, barge‑in interruption, multi‑LLM support, JWT auth, conversation memory summarisation, and optional face‑restoration via GFPGAN.

  • Project
    Soul-AILab/SoulX-FlashTalk

    SoulX-FlashTalk is a system for real-time, infinite streaming of audio-driven avatars using self-correcting bidirectional distillation to create continuous talking head animations.

  • Project
    anliyuan/Ultralight-Digital-Human

    A lightweight digital human model designed for real-time talking-head generation on mobile devices using personalized training from short videos.