paninski-lab/lightning-pose

Accelerated 2D and 3D pose estimation using semi-supervised learning

What it solves

Lightning Pose is designed for robust animal pose estimation, addressing the challenge of occlusions in both single-view and multi-view video feeds. It provides an end-to-end workflow for tracking animal movements with high accuracy even when parts of the animal are unable to be seen clearly.

How it works

The project uses advanced transformer architectures. For multi-view setups, it employs Multi-View Transformers and simulated occlusions to learn geometric relationships between different camera angles. For single-view datasets, it leverages temporal context and unsupervised losses to maintain performance in challenging scenarios.

Who it’s for

It is primarily aimed at researchers and scientists who need to track animal poses in video data, supporting both single and multi-view camera configurations.

Highlights

  • End-to-end package including a GUI for labeling, model management, and evaluation.
  • Support for both single-view and multi-view animal pose estimation.
  • High performance on occlusions using transformer-based architectures.
  • Compatible with the Ensemble Kalman Smoother (EKS) for post-processing.
  • Fast training times, with typical backbones like ResNet-50 or ViT-Small fitting on ~200 labeled frames in about 20 minutes on a T4 GPU.

Related

  • Project
    phongdaot/MocapAnything

    An end-to-end motion capture system that converts monocular videos into 3D joint rotations for arbitrary skeletons, enabling movement transfer to any rigged character.

  • Project
    DeepLabCut/DeepLabCut

    DeepLabCut is a markerless pose estimation toolbox that allows researchers to track animals and objects in video without physical markers using deep learning.

  • Project
    talmolab/sleap

    A deep-learning framework for multi-animal pose tracking that uses a human-in-the-loop GUI to rapidly label and quantify animal behavior.

  • Project
    zhouhengamerica/XLens

    X-Lens is a real-time metric depth estimation system that can process a mix of pinhole and fisheye cameras simultaneously using a single model to generate a fused 3D point cloud.

  • Project
    yzfzzz/depth-detect

    A high-performance C++ and TensorRT framework that fuses YOLO object detection with monocular/binocular depth estimation to track objects and estimate their motion states.