apple-aiml-research/ml-neuman

What it solves

NeuMan addresses the challenge of reconstructing both a static background scene and a dynamic, animatable human figure from a single input video. It allows for the synthesis of novel views and novel poses of the person within the original environment.

How it works

The system uses Neural Radiance Fields (NeRF) to represent the scene. It separates the reconstruction into two parts: a background scene NeRF and a human NeRF. The human model is integrated with SMPL (a parametric human body model) to allow for animation and reposing, enabling the system to render the person in poses not seen in the original video.

Who it’s for

This project is for researchers and developers working in computer vision, 3D reconstruction, and neural rendering, specifically those interested in human-centric scene synthesis.

Highlights

• Single Video Input: Reconstructs a full 3D scene and person from just one video.
• Animatable Humans: Supports novel pose synthesis using SMPL weights.
• Compositional Synthesis: Capable of rendering multiple reconstructed actors into a single scene (telegathering).
• Novel View Synthesis: Generates 360-degree views of both canonical and posed humans.