phongdaot/MocapAnything

[SIGGRAPH ASIA 2026]: End-to-End Motion Capture for Arbitrary Skeletons

What it solves

MoCapAnything V2 enables end-to-end motion capture from a single monocular video, converting it into 3D joint rotations ready for animation (BVH format). Unlike many motion capture tools, it is not limited to human bodies; it supports arbitrary skeletons, including animals, objects, and custom rigged characters, without requiring a template-specific body model.

How it works

The system uses a main model called video2pose2rot, which is a single end-to-end network. It maps a video sequence and a reference skeleton to joint rotations through two jointly fine-tuned subtasks:

  1. video2pose: Predicts 3D joint positions from the image sequence and a reference pose.
  2. pose2rot: Converts those joint positions into joint rotations based on a reference pose-rotation pair.

This approach is mesh-free, avoiding the need for analytical Inverse Kinematics (IK) or complex mesh-based pipelines, making it approximately 20— faster than mesh-based alternatives.

Who it’s for

This tool is designed for animators, 3D artists, and researchers who need to transfer motion from real-world videos to various 3D characters (humans, animals, or custom rigs) without expensive motion capture equipment.

Highlights

  • Arbitrary Skeleton Support: Works with animals, humans, and custom FBX rigged characters.
  • End-to-End Pipeline: Directly predicts BVH-ready rotations from video to pose to rotation.
  • High Performance: Significantly faster than mesh-based pipelines by predicting joints directly.
  • Custom Rig Integration: Allows users to bring their own rigged FBX files to generate skeleton topology and reference poses.

Related

  • Project
    nvidia-isaac/video_to_data

    Video to Data (V2D) is a modular, container‑based pipeline that converts ordinary human‑demo videos into 3‑D reconstruction assets and physics‑grounded robot RL policies, using LangGraph‑driven video ingestion, a suite of vision modules (depth, masks, meshes, SMPL), and NVIDIA Isaac Lab for training.

  • Project
    zai-org/SCAIL-2

    SCAIL-2 is an open-source model for end-to-end character animation and replacement, removing the need for intermediate pose representations to support complex motions and diverse identities.

  • Project
    FORTH-ModelBasedTracker/MocapNET

    A real-time 2D-to-3D human pose estimator that converts RGB video feeds into standard BVH motion capture files for 3D animation.

  • Project
    perfanalytics/pose2sim

    An open-source workflow for markerless 3D kinematics that converts 2D pose estimation from standard cameras into research-grade 3D joint angles and skeletal analysis.

  • Project
    paninski-lab/lightning-pose

    An end-to-end animal pose estimation package that uses transformer architectures to robustly track movements in single-view and multi-view videos, especially during occlusions.