zju3dv/animatable_nerf

What it solves

This project addresses the challenge of creating realistic, animatable 3D digital avatars of humans from videos. It allows for the reconstruction of a human body that can be posed in new, unseen configurations while maintaining visual fidelity from multiple viewing angles.

How it works

The system uses implicit neural representations to model the human body. It provides several implementation variants:

• Animatable NeRF: The original version using Neural Radiance Fields to model volume density and color.
• NeRF-PDF: A version where the neural blend weight field is replaced with a pose-dependent displacement field.
• SDF-PDF (Animatable Neural Fields/SDF): A version that replaces the canonical NeRF model with a neural surface field, outputting a Signed Distance Function (SDF) instead of volume density, also utilizing a displacement field.

These models are trained on multi-view video data (such as Human3.6M or ZJU-MoCap) to learn a canonical representation of the person that can be deformed to match specific human poses.

Who it’s for

This is primarily for researchers and developers in computer vision and graphics focusing on human body modeling, digital human synthesis, and neural rendering.

Highlights

• Supports multiple representation types including vanilla NeRF and SDF-based neural surface fields.
• Capable of generating novel views and animating humans into unseen poses.
• Includes tools for generating posed 3D meshes from the learned representations.
• Evaluated on several datasets including Human3.6M, ZJU-MoCap, and SyntheticHuman++.