Francis-Rings/StableAvatar

StableAvatar – Infinite‑Length Audio‑Driven Avatar Video Generation

What it is

• A research‑grade video diffusion model that creates talking‑head/avatar videos from a single reference image and an audio track.
• Unlike prior audio‑driven generators, StableAvatar can produce unbounded video length while keeping the subject’s identity and lip‑sync accurate, without any post‑processing (no face‑swap or restoration steps).

Key technical ideas

Component	Purpose
Time‑step‑aware Audio Adapter	Modulates audio embeddings per diffusion timestep to stop the latent drift that normally breaks long‑clip generation.
Audio Native Guidance	Uses the model’s own evolving audio‑latent prediction as a dynamic guide, tightening lip‑sync during inference.
Dynamic Weighted Sliding‑window	Overlaps consecutive generation windows and blends latents, yielding smooth transitions across an arbitrarily long video.
Video Diffusion Transformer (DiT) backbone	A 1.3 B‑parameter transformer (Wan2.1) trained on mixed‑resolution talking‑face datasets (512×512, 480×832, 832×480).

What you can do with it

• Generate endless avatar clips at three resolutions (512×512, 480×832, 832×480) from any portrait image and speech/audio file.
• Fine‑tune the model or train LoRA adapters for custom identities or styles (training scripts and LoRA code are provided).
• Run inference on a single GPU (≈3 GB memory with CPU off‑load) or scale to multi‑GPU setups using the provided multiple_gpu_inference.sh script.
• Plug the model into Gradio, ComfyUI, or Hugging Face Spaces for interactive demos.

Quickstart summary

1. Install the exact PyTorch 2.6 (or 2.7 for Blackwell GPUs) and other deps from requirements.txt.
2. Download the checkpoint from the Hugging Face repo (FrancisRing/StableAvatar).
3. Prepare an audio file (optional vocal‑separation with audio-separator[gpu]).
4. Run bash inference.sh (or python app.py for a Gradio UI). Adjust --width, --height, --sample_steps, --overlap_window_length, and guidance scales to trade quality vs speed.
5. Combine the silent output video with the original audio via ffmpeg if you need an MP4 with sound.

Training data layout

• The repo expects a folder talking_face_data/ with three resolution branches (square, vec, rec).
• Each branch contains sub‑folders for speech, singing, and dancing; inside are per‑clip directories with sub_clip.mp4, audio.wav, images/, face_masks/, and lip_masks/.
• Scripts are provided to extract frames, vocal tracks, face masks (via the companion StableAnimator repo), and lip masks (mediapipe).

Hardware notes

• Basic inference fits ~3 GB GPU memory using sequential_cpu_offload.
• For faster generation, flash_attn can be installed.
• Multi‑GPU inference uses the --ulysses_degree × --ring_degree scheme; FSDP can be enabled with --fsdp_dit.

Resources

• Project page, paper (arXiv 2508.08248), Hugging Face model hub, demo Space, YouTube/Bilibili videos.
• The code is open‑source under the repository’s license (see LICENSE).

Who might use it

• Researchers exploring long‑form audio‑driven video synthesis.
• Creators who need high‑fidelity talking‑head avatars for virtual presenters, games, or education.
• Developers building UI tools (Gradio, ComfyUI) that require a plug‑and‑play avatar generator.