facebookresearch/spider

What it solves

SPIDER provides a general framework for physics-based retargeting, allowing motions from human videos to be transferred to diverse robot embodiments, including dexterous hands and humanoid robots. It addresses the challenge of mapping human movements to robot hardware that may have different physical structures and constraints.

How it works

The framework uses a physics-informed pipeline to convert human video data into robot actions. The process involves several steps: processing raw datasets into a unified schema, decomposing objects, detecting contacts, generating scene XMLs, and performing kinematic and physics-based retargeting. It supports multiple simulators (such as Mujoco Wrap, Genesis, and IsaacGym) and integrates with various downstream training pipelines for reinforcement learning (RL) and behavioral cloning (BC).

Who it’s for

This tool is designed for robotics researchers and developers working on dexterous manipulation and humanoid robot control, specifically those looking to translate human demonstrations into executable robot trajectories.

Highlights

• Broad Embodiment Support: Supports over 9 robots and 6 datasets out of the box.
• Physics-Based: The first general physics-based retargeting pipeline for both dexterous hands and humanoid robots.
• Sim2Real Ready: Designed for deployment to real-world robots.
• Flexible Integration: Native support for multiple simulators and seamless integration with RL training and data augmentation.