GVCLab/PersonaLive

What it solves

PersonaLive is designed for real-time, streamable portrait animation. It allows a single reference image of a person to be animated by a driving video, enabling the creation of infinite-length portrait animations suitable for live streaming applications.

How it works

The project uses a diffusion framework that integrates several components to translate motion from a driving video to a reference image. It employs a motion encoder, a pose guider, and a temporal module to ensure smooth and expressive animations. To achieve real-time performance, it supports a streaming generation strategy and can be accelerated using TensorRT, which can roughly double the inference speed.

Who it’s for

This tool is primarily for academic researchers and developers interested in portrait animation, digital humans, and real-time video generation.

Highlights

• Real-time Streaming: Capable of generating infinite-length animations with low latency.
• Hardware Optimization: Supports TensorRT acceleration and xFormers for memory efficiency.
• VRAM Efficiency: Includes a streaming strategy that allows long video generation on GPUs with as little as 12GB VRAM.
• Comprehensive Training Pipeline: Provides a three-stage training workflow (image-level warm-up, adversarial refinement, and temporal fine-tuning).
• Integration: Supported by ComfyUI via a community plugin.