zju3dv/EasyMocap

What it solves

EasyMocap provides a comprehensive toolbox for markerless human motion capture and novel view synthesis. It eliminates the need for physical markers on subjects, allowing users to extract 3D body, hand, and face poses from various video sources, including multi-view setups, single internet videos, and videos featuring mirrors.

How it works

The system fits parametric human models (such as SMPL, SMPL-X, and MANO) to video data. Depending on the source, it uses different techniques:

• Multi-view: Fits models to synchronized cameras to capture detailed poses.
• Internet Videos: Combines 2D keypoint estimation (via tools like MediaPipe or HRNet) and CNN initialization to reconstruct 3D motion.
• Mirrored Videos: Specifically handles the geometry of humans appearing in mirrors to reconstruct 3D poses.
• Novel View Synthesis: Uses implicit neural representations to generate new viewpoints of dynamic humans from sparse views.

Who it’s for

This tool is designed for researchers and developers in 3D vision, computer graphics, and animation who need to capture human movement from RGB videos without expensive marker-based hardware.

Highlights

• Versatile Input: Supports multi-view, single-view internet videos, and mirrored videos.
• Comprehensive Model Support: Compatible with SMPL, SMPL+H, SMPL-X, and MANO models for full-body, hand, and face capture.
• Novel View Synthesis: Ability to synthesize new views of human interactions from sparse multi-view videos.
• Integrated Tooling: Includes built-in tools for camera calibration, 3D real-time visualization, and an annotator for bounding boxes and keypoints.