jdh-algo/JoyVASA

Diffusion-based Portrait and Animal Animation

What it solves

JoyVASA addresses inefficiencies in audio-driven portrait animation, such as high training/inference costs and limitations on video length and inter-frame continuity. It specifically enables the animation of both human and animal faces from a single reference image and an audio clip, ensuring high-quality lip-sync and natural head motion.

How it works

The system uses a two-stage diffusion-based process:

  1. Decoupled Representation: It separates dynamic facial expressions from static 3D facial representations. This allows the model to combine any static face with various motion sequences to create longer videos.
  2. Motion Generation: A diffusion transformer generates identity-independent motion sequences directly from audio cues (using encoders like wav2vec2 or Hubert).
  3. Rendering: A generator takes the 3D facial appearance and the generated motion sequences to render the final high-quality animation.

Who it’s for

This project is designed for researchers and developers working on AI-driven video generation, digital humans, and animal animation who need a flexible, audio-driven facial animation framework.

Highlights

  • Cross-Species Animation: Capable of animating both human portraits and animal images.
  • Decoupled Framework: Separates identity from motion, enabling longer video generation and better continuity.
  • Multilingual Support: Trained on a hybrid dataset of Chinese and English data.
  • Customizable Expressions: Includes a cfg_scale parameter to adjust the intensity of expressions and poses.