phongdaot/MocapAnything

[SIGGRAPH ASIA 2026]: End-to-End Motion Capture for Arbitrary Skeletons

What it solves

MoCapAnything V2 enables end-to-end motion capture from a single monocular video for virtually any skeleton type, allowing users to transfer movement from a video (such as a human dancing) to a target 3D asset (such as an animal or custom rigged character) without needing complex manual setup or specialized hardware.

How it works

The project uses a main model called video2pose2rot, which maps video sequences directly to joint rotations (BVH format). It combines two sub-tasks: video2pose (predicting joint positions from images) and pose2rot (converting those positions into rotations). To handle different species and scales, it uses a single reference pose-rotation pair from the target asset to anchor the coordinate system. This approach removes the need for analytical Inverse Kinematics (IK) and is significantly faster than previous mesh-based methods.

Who it’s for

This tool is designed for 3D animators, game developers, and researchers who need to drive rigged characters with real-world footage, specifically those working with non-human skeletons or custom rigs.

Highlights

  • End-to-End Pipeline: Directly converts video to joint rotations without requiring an analytical IK loop.
  • Arbitrary Skeleton Support: Generalizes to unseen animals and custom rigged FBX characters.
  • High Performance: Approximately 20— faster than mesh-based pipelines because it is mesh-free.
  • Dance Anything Feature: Integrates SAM2 to isolate a dancer from a video and retarget those movements to a target creature, complete with original audio sync.

Related

  • Project
    FORTH-ModelBasedTracker/MocapNET

    A real-time 2D-to-3D human pose estimator that converts RGB video feeds into standard BVH motion capture files for 3D animation.

  • Project
    perfanalytics/pose2sim

    A markerless 3D kinematics workflow that converts 2D video from low-cost cameras into research-grade 3D joint angles and skeletal analysis using OpenSim.

  • Project
    zai-org/SCAIL-2

    SCAIL-2 is an open-source model for end-to-end character animation and replacement, removing the need for intermediate pose representations to support complex motions and diverse identities.

  • Project
    paninski-lab/lightning-pose

    An end-to-end animal pose estimation package that uses transformer architectures to robustly track movements in single-view and multi-view videos, especially during occlusions.

  • Project
    zju3dv/MatchAnything

    MatchAnything is a research project that trains a single neural network to find correspondences between images from any visual modality (e.g., RGB, infrared, sketches). Pre‑trained weights are hosted on Hugging Face, and an online demo lets users try cross‑modality matching instantly. The approach aims to replace many specialized matching pipelines with one universal model, useful for robotics, remote sensing, medical imaging, and AR.