NVlabs/GEM-X

What it solves

GEM-X addresses the need for accurate 3D human pose estimation from images and videos, combining body pose, hand pose, and temporal body motion in one unified framework. Traditional approaches often require separate models and pipelines; GEM-X provides a single codebase for inference, training, and evaluation across these tasks, reducing complexity for applications like robotics, AR/VR, and video analysis.

How it works

The framework consists of three models: Body Pose and Hand Pose reconstruct 3D geometry from single images, while Body Motion uses temporal context to estimate camera-space and world-space human motion from monocular video. All models rely on SOMA geometry. There are four inference modes: pose-image, pose-video, motion-video, and motion-cache, which handle different input types and processing needs. Training and fine-tuning workflows are shared across tasks, with pretrained checkpoints available for each model. The pipeline includes optional components like person detection, hand tracking, and contact/IK postprocessing.

Who it’s for

Researchers and developers working on human pose estimation, robotics (including motion retargeting to robots like Unitree G1), AR/VR, video understanding, and production deployment of pose estimation systems. The README also highlights integration with NVIDIA AI for Media for scalable deployment.

Highlights

• Unified framework for body pose, hand pose, and body motion with three models and four inference modes.
• Single-image body and hand reconstruction, plus temporal body motion from monocular video.
• Flexible hand inputs: supplied boxes or RT-DETR detection, ego/exo processing profiles, tracking and smoothing.
• Robotics showcase: motion retargeting to Unitree G1 using SOMA Retargeter.
• Training and fine-tuning supported for all three models, with pretrained weights for each.
• Evaluation with standard benchmarks (MPJPE, PA-MPJPE, PVE) and native hand metrics.
• ONNX/TensorRT acceleration added (as of 2025-06).
• Apache 2.0 license with per-file exceptions.