FORTH-ModelBasedTracker/MocapNET

A real-time method that estimates the 3D human pose directly in the popular Bio Vision Hierarchy (BVH) format, given estimations of the 2D body joints originating from monocular color images. Our contributions include: (a) A novel and compact 2D pose NSRM representation. (b) A human body orientation classifier and an ensemble of orientation-tuned

What it solves

MocapNET is designed to solve the problem of estimating 3D human poses from monocular RGB images (single-camera video) in real-time. It converts 2D joint estimations into the Bio Vision Hierarchy (BVH) format, which is a standard for 3D animation, allowing users to create 3D animations from simple video feeds without expensive motion capture hardware.

How it works

The system operates as a 2D-to-3D pose estimator. It first uses a 2D joint estimator (such as MobileNet, OpenPose, or Mediapipe) to detect body joints in a video frame. These 2D coordinates are then processed by an ensemble of orientation-tuned neural networks that regress the 3D pose.

Key technical components include:

  • NSRM Representation: A compact 2D pose representation.
  • Orientation Classifier: A system that identifies body orientation to choose the appropriate neural network for 3D regression.
  • Inverse Kinematics (IK) Solver: An efficient solver that refines the 3D output to ensure the pose is consistent with the limb sizes of a specific target person.
  • Blender Integration: A plugin that allows the output BVH files to be used with custom skinned humans in the Blender 3D editor.

Who it’s for

This project is for 3D animators, researchers in computer vision, and developers building applications for motion capture, automotive 3D body tracking, or virtual avatars.

Highlights

  • Real-time Performance: Capable of running at 30-70 FPS depending on the hardware and configuration.
  • Holistic Tracking: Version 4.0 supports tracking of the body, hands, and face, including 3D gaze and BVH facial configurations.
  • BVH Output: Directly outputs motion frames in a standard format compatible with most 3D engines.
  • Occlusion Robustness: Uses a decomposed kinematic hierarchy to recover poses even when parts of the body are occluded.