zju3dv/GVHMR

GVHMR – World‑Grounded Human Motion Recovery

What it is – GVHMR is a research codebase that reconstructs 3‑D human motion from a single video while keeping the motion anchored to the real world (gravity‑aligned coordinates). It builds on visual‑odometry (DPVO or a lightweight SimpleVO) to estimate camera motion and then lifts 2‑D pose detections into a globally consistent 3‑D trajectory.

Key components

• Gravity‑View Coordinates – a novel representation that aligns the recovered motion with the physical gravity direction, making the output directly usable for downstream tasks such as animation, AR/VR, or robotics.
• Visual odometry integration – optional DPVO (dense‑pixel VO) or the newer SimpleVO for estimating camera pose when the camera moves.
• Training & evaluation scripts – ready‑to‑run pipelines for the 3DPW, RICH, and EMDB datasets, matching the results reported in the SIGGRAPH Asia 2024 paper.
• Demo utilities – simple command‑line tools (tools/demo/demo.py, tools/demo/demo_folder.py) and ready‑made Colab/HuggingFace demos for quick inference on arbitrary videos.

Getting started

1. Installation – follow the step‑by‑step guide in docs/INSTALL.md (Python 3.9+, PyTorch, CUDA, and a few vision libraries). The repo provides a requirements.txt‑style list.
2. Run a demo –

   python tools/demo/demo.py --video=docs/example_video/tennis.mp4 -s   # -s skips VO if the camera is static
   

   Use demo_folder.py to batch‑process a directory of videos.
3. Reproduce paper results – a single command evaluates on all three benchmark datasets:

   python tools/train.py global/task=gvhmr/test_3dpw_emdb_rich \
       exp=gvhmr/mixed/mixed \
       ckpt_path=inputs/checkpoints/gvhmr/gvhmr_siga24_release.ckpt
   

4. Training – start from the released checkpoint or train from scratch (the released model was trained on two RTX 4090 GPUs for 420 epochs). Example:

   python tools/train.py exp=gvhmr/mixed/mixed
   

   Note: training does not include the post‑processing used in the test script, so raw metrics will differ slightly.

Recent updates

• 2025‑03‑08 – added a lightweight SimpleVO (no DPVO dependency) and a f_mm flag to set the full‑frame camera focal length in millimetres.

Where to see it in action

• Colab demo: https://colab.research.google.com/drive/1N9WSchizHv2bfQqkE9Wuiegw_OT7mtGj
• HuggingFace Space: https://huggingface.co/spaces/LittleFrog/GVHMR (interactive web UI).

Citation

@inproceedings{shen2024gvhmr,
  title={World-Grounded Human Motion Recovery via Gravity-View Coordinates},
  author={Shen, Zehong and Pi, Huaijin and Xia, Yan and Cen, Zhi and Peng, Sida and Hu, Zechen and Bao, Hujun and Hu, Ruizhen and Zhou, Xiaowei},
  booktitle={SIGGRAPH Asia Conference Proceedings},
  year={2024}
}

Who built it – The project comes from the ZJU‑3DV group (Zhejiang University) and collaborators, and acknowledges prior work such as WHAM, 4D‑Humans, and ViTPose‑Pytorch.