zju3dv/neuralbody

Neural Body – Implicit Neural Representations for Dynamic Humans

What it is

• A research codebase that learns a continuous 3D representation of a moving human body from multi‑view video. It uses implicit neural fields (a neural network that maps 3‑D coordinates to color/occupancy) together with structured latent codes that encode pose‑dependent geometry.
• The goal is novel‑view synthesis: given a few calibrated cameras, the model can render the person from any new viewpoint, even for poses that were not seen during training.

Key capabilities

• Training on two public datasets:
  • ZJU‑MoCap – a multi‑view capture of a single actor performing many motions.
  • People‑Snapshot – a collection of short videos of different people.
• Inference for:
  • Rendering novel views of a single frame.
  • Rendering a video of a moving person from a fixed or rotating camera.
  • Extracting a mesh of the body at any frame.
• Supports unseen poses (generalisation to new motions) and white‑background rendering.
• Distributed training via PyTorch’s torch.distributed and a ready‑to‑use Docker image.

How to get started

1. Install – follow INSTALL.md for a manual conda setup or use the Dockerfile in docker/ (thanks to Zhaoyi Wan).
2. Download data – the README links to scripts that fetch the ZJU‑MoCap and People‑Snapshot datasets, plus pretrained checkpoints hosted on Google Drive.
3. Run examples – the repo ships ready‑made shell scripts:
   • visualize.sh shows how to render novel views, pose‑varying videos, and meshes.
   • train.sh and test.sh illustrate single‑GPU and multi‑GPU training/evaluation commands.
4. Custom data – a tools/custom folder explains how to adapt the pipeline to your own multi‑view capture.

Datasets & resources

• ZJU‑MoCap – multi‑view capture with SMPL parameters (fitted with EasyMocap). The repo provides both the original and a newer set of SMPL fits.
• People‑Snapshot – in‑the‑wild videos; the repo includes processing scripts (process_snapshot.py) and a visualizer for the SMPL parameters.
• Mobile‑Stage and SyntheticHuman++ datasets were announced in Jan 2024 (links to Google Forms for access).

Typical workflow

# 1. Prepare data (e.g., People‑Snapshot)
python tools/process_snapshot.py --input path/to/video

# 2. Train (single GPU)
python train_net.py --cfg_file configs/snapshot_exp/snapshot_f3c.yaml \
    exp_name female3c resume False

# 3. Visualize novel view of a frame
python run.py --type visualize \
    --cfg_file configs/snapshot_exp/snapshot_f3c.yaml \
    exp_name female3c vis_novel_view True num_render_views 144

# 4. Render a mesh
python run.py --type visualize \
    --cfg_file configs/snapshot_exp/snapshot_f3c.yaml \
    exp_name female3c vis_mesh True

Who might use it

• Researchers studying neural rendering, human shape/pose modeling, or novel‑view synthesis.
• Developers needing a baseline for generating photorealistic avatars from sparse multi‑camera setups.
• Anyone interested in extracting animatable 3‑D meshes from video.

Citation If you use the code or the pretrained models, cite the CVPR 2021 and TPAMI 2023 papers (BibTeX provided in the README).

---

All information above is taken directly from the repository’s README; no external assumptions have been added.