Alibaba-Quark/LiveAvatar

Live Avatar – Real‑time, Infinite‑length Audio‑driven Avatar Generation

Live Avatar is a research‑grade system that turns a spoken audio track (and an optional reference image / text prompt) into a continuous video of a talking avatar. The core is a 14 B‑parameter diffusion model (WanS2V‑14B) fine‑tuned with a LoRA checkpoint (Live‑Avatar). By combining a custom block‑wise autoregressive pipeline with aggressive inference tricks (4‑step sampling, FlashAttention, FP8 quantisation, compilation), the authors achieve ≈45 FPS streaming generation on a multi‑GPU H800 setup and can keep generating for 10 000 + seconds without restarting.

---

Key Features

Feature	What it means
Real‑time streaming	Generates video frames on‑the‑fly while audio is being streamed, with low latency (≈45 FPS).
Infinite‑length autoregressive	The pipeline processes video in blocks, allowing arbitrarily long streams (tested > 10 000 s).
Audio‑driven	Input is a waveform; the model synchronises lip movements and facial expressions to the speech.
Reference image & optional text	Users can supply a portrait (or cartoon) to define the avatar’s appearance and a prompt to steer style.
14 B diffusion backbone	Built on the WanS2V‑14B video diffusion model, providing high visual fidelity.
Speed‑up tricks	4‑step DDPM sampling, FlashAttention 2/3, pipeline parallelism, optional FP8 quantisation, and JIT compilation.
Multi‑GPU and single‑GPU support	Official scripts for 5‑GPU streaming inference and for a single 80 GB GPU (offline generation).
Open‑source components	Model weights on Hugging Face, demo website, Gradio UI, and inference scripts are released under Apache 2.0.

---

Quick Start (Linux, Conda)

# 1️⃣ Create environment
conda create -n liveavatar python=3.10 -y
conda activate liveavatar

# 2️⃣ (Optional) Install CUDA 12.4 toolkit
conda install nvidia/label/cuda-12.4.1::cuda -y
conda install -c nvidia/label/cuda-12.4.1 cudatoolkit -y

# 3️⃣ Install PyTorch + FlashAttention
pip install torch==2.8.0 torchvision==0.23.0 \
    --index-url https://download.pytorch.org/whl/cu128
# For Hopper GPUs (H800/H200) use FlashAttention‑3, otherwise FlashAttention‑2
pip install flash_attn_3 --find-links https://windreamer.github.io/flash-attention3-wheels/cu128_torch280 --extra-index-url https://download.pytorch.org/whl/cu128
# or
# pip install flash-attn==2.8.3 --no-build-isolation

# 4️⃣ Install Python deps
pip install -r requirements.txt

# 5️⃣ Install ffmpeg (Ubuntu example)
sudo apt-get update && sudo apt-get install -y ffmpeg

# 6️⃣ Download model checkpoints (replace with your HF token if needed)
export HF_ENDPOINT=https://hf-mirror.com   # optional for China mainland
pip install "huggingface_hub[cli]"
huggingface-cli download Wan-AI/Wan2.2-S2V-14B --local-dir ./ckpt/Wan2.2-S2V-14B
huggingface-cli download Quark-Vision/Live-Avatar --local-dir ./ckpt/LiveAvatar

---

Running Inference

Multi‑GPU streaming (recommended for full speed)

bash infinite_inference_multi_gpu.sh   # CLI version
# or
bash gradio_multi_gpu.sh               # launches a Gradio UI

Requires at least five GPUs with ≥ 80 GB VRAM each.

Single‑GPU (offline) generation

bash infinite_inference_single_gpu.sh   # CLI version
# or
bash gradio_single_gpu.sh               # Gradio UI

Works on a single 80 GB GPU; may need to lower size to avoid OOM.

Important flags

• ENABLE_COMPILE=true – activates JIT compilation; first run is slow but later runs are much faster (crucial for long streams).
• ENABLE_FP8=true – switches to FP8 quantisation, cutting VRAM usage to fit 48 GB GPUs at a modest quality loss.
• --num_clip N – generate N short clips for quick preview.
• --enable_online_decode false – disables on‑the‑fly decoding on single‑GPU runs to reduce CPU‑GPU traffic (may slightly affect quality for very long videos).

---

What the Repository Contains

• Inference scripts (infinite_inference_*.sh, gradio_*.sh) that set up the TPP (Temporal‑Parallel‑Pipeline) and launch generation.
• requirements.txt with exact Python package versions used by the authors.
• Model checkpoints are hosted on Hugging Face; the repo only includes download helpers.
• Demo page (https://liveavatar.github.io/) showcasing human‑AI conversation, infinite video loops, and character diversity.
• Citation and license (Apache 2.0) information.

---

Roadmap (as listed in the repo)

• ✅ Multi‑GPU real‑time streaming (already released)
• ✅ Single‑GPU offline inference
• ✅ FP8 quantisation and compilation speed‑ups
• ⬜ UI integration for effortless streaming interaction
• ⬜ Text‑to‑speech (TTS) coupling
• ⬜ Full training pipeline (future release)
• ⬜ Further model optimisation (RoPE, LoRA merge, streaming‑VAE integration)

---

License & Acknowledgements

• The code and most assets are Apache 2.0.
• The base diffusion model (WanS2V‑14B) is also Apache 2.0.
• The authors thank related works such as CausVid, LongLive, and WanS2V.

---

Who Might Use This?

• Researchers exploring audio‑driven video synthesis, streaming diffusion, or low‑latency generative AI.
• Developers building interactive avatars for virtual assistants, live streaming, or metaverse experiences.
• Artists who need a controllable, real‑time tool to animate characters from voice recordings.

---

Bottom Line

Live Avatar demonstrates that a diffusion‑based video model can be engineered to run real‑time, block‑wise, and indefinitely on modern GPU clusters, opening the door to live, AI‑driven avatar experiences.