anliyuan/Ultralight-Digital-Human

What it solves

This project provides a lightweight digital human model capable of real-time performance on mobile devices. It enables the creation of a personalized talking-head avatar from a short video (3-5 minutes) of a specific person, synchronizing lip movements to a provided audio track.

How it works

The system uses a personalized training approach where a dedicated model is trained for each individual. It processes audio using encoders like HuBERT (for higher quality) or Wenet (for faster, mobile-ready inference). A UNet-based architecture then maps these audio features to visual facial movements. The project supports streaming inference to reduce latency and allows for ONNX export to optimize performance on mobile hardware.

Who it’s for

• Developers building real-time AI avatars for mobile applications.
• Users wanting to create a personalized digital double with minimal training data.
• Researchers interested in lightweight, on-device talking-head generation.

Highlights

• Mobile Real-time Performance: Designed specifically to run efficiently on mobile devices.
• Flexible Audio Encoders: Supports both HuBERT for quality and Wenet for speed.
• Streaming Support: Includes logic for streaming inference to achieve low-latency playback.
• Personalized Training: Simple pipeline to train a custom model from a few minutes of video footage.