perfanalytics/pose2sim

Markerless kinematics with any cameras — From 2D Pose estimation to 3D OpenSim motion

What it solves

Pose2Sim provides a markerless 3D kinematics workflow for humans and animals, serving as a low-cost alternative to traditional marker-based Motion Capture (MoCap) systems. It allows for research-grade 3D motion analysis using standard hardware like smartphones, webcams, or GoPros, enabling capture in diverse environments such as sports fields, clinics, or outdoors without requiring subjects to wear markers.

How it works

The system converts 2D pose estimation data into 3D skeletal analysis. The workflow consists of several stages:

  1. Pose Estimation: Extracts 2D keypoints using models like RTMPose (default), DeepLabCut, or legacy models like OpenPose.
  2. Calibration & Sync: Synchronizes multiple camera views and calculates intrinsic and extrinsic camera parameters.
  3. Triangulation: Associates people across different camera views and triangulates 2D keypoints into 3D coordinates.
  4. Refinement: Filters 3D coordinates and augments markers to improve data quality.
  5. Kinematics: Computes full-body 3D joint angles and skeletal analysis, which can be integrated with OpenSim for visualization and analysis.

Who it’s for

  • Sports Analysts: For field-based 3D motion capture.
  • Clinicians: For gait analysis and clinical assessments in a medical office.
  • Animators: For outdoor 3D capture of fully clothed subjects.
  • Biomechanics Researchers: For conducting studies with multiple participants using accessible hardware.

Highlights

  • High Accuracy: Validated by peer-reviewed studies with a joint angle error of 2-6°.
  • Hardware Agnostic: Compatible with any combination of phones, webcams, or GoPros.
  • Multi-Person Support: Capable of tracking multiple people simultaneously across views.
  • Privacy-Focused: Runs locally to comply with data privacy and GDPR rules.
  • OpenSim Integration: Produces .trc and .mot files for full skeletal analysis in OpenSim or Blender.

Related

  • Project
    opencap-org/opencap-core

    A pipeline that estimates 3D human movement kinematics and marker positions from multiple smartphone videos using OpenPose and OpenSim.

  • Project
    davidpagnon/Sports2D

    A tool for automatic 2D motion capture and biomechanical analysis that computes joint trajectories and angles from a single video or webcam.

  • Project
    stereolabs/zed-ros2-wrapper

    Official ROS 2 wrapper for Stereolabs ZED depth cameras, providing image, depth, point‑cloud, SLAM, object detection, body tracking, GNSS fusion and simulation support. Installable on Ubuntu with ZED SDK and CUDA, launchable via `ros2 launch zed_wrapper zed_camera.launch.py`.

  • Project
    Tau-J/rtmlib

    rtmlib is a lightweight Python library for human and animal pose estimation that wraps RTMPose and ViTPose models. It avoids heavy OpenMMLab dependencies, requiring only numpy, OpenCV and ONNX Runtime (with optional GPU/accelerator back‑ends). The package provides high‑level solution classes (Wholebody, Body, Hand, Animal, Custom, Wholebody3d) that combine a detector (YOLOX, RFDETR, RTMDet) and a pose estimator, plus utilities for drawing skeletons. Models are auto‑downloaded from OpenMMLab or a HuggingFace mirror. Install via pip or from source, and run a Gradio WebUI or use the Python API for single‑image, video, or webcam inference.

  • Project
    emilianavt/OpenSeeFace

    OpenSeeFace is an open‑source, CPU‑only facial‑landmark tracker (66 points) built on a MobileNetV3 model exported to ONNX. It streams landmark, gaze, and blink data over UDP for real‑time avatar animation in Unity, Godot, VSeeFace, etc. Multiple model sizes let you balance speed (up to 213 fps) against accuracy. The project includes Unity components for receiving data, visualising points, driving IK, and training a small SVM‑based expression classifier. It works robustly under low light, noise, and partial occlusion, but eye‑region precision is modest and tracking a single face at 30 fps can consume an entire CPU core. Licensed under BSD‑2‑clause.