apple-aiml-research/ml-hugs

What it solves

HUGS addresses the challenge of reconstructing both a static background scene and an animatable human figure from a single video source. It allows for the creation of a 3D representation that can be manipulated and animated into new poses while maintaining the scene's context.

How it works

The system uses neural radiance fields and Gaussian Splatting. It can be trained in three different modes: jointly optimizing both the human and the scene, focusing only on the human using a Triplane+MLP model, or focusing only on the scene. It leverages the SMPL body model to handle human geometry and can be integrated with datasets like AMASS for rendering novel animations.

Who it’s for

This project is designed for researchers and developers in computer vision and 3D reconstruction who want to create animatable digital humans within a reconstructed environment.

Highlights

• Reconstructs both human and background from a single video.
• Supports novel animation rendering using external motion capture data.
• Flexible training modes for joint, human-only, or scene-only optimization.
• Implements a Triplane+MLP model for human representation.