malik-group/do-as-i-do

What it solves

Do as I Do is designed to help robots learn from human human-object interaction videos. It solves the problem of translating a human's hand-object interaction from a video demo into a trajectory that a robot can actually execute on physical hardware.

How it works

The project uses a three-stage pipeline:

1. Reconstruction: It converts a video of a human hand and objects into 3D meshes, pointmaps, and 6-DoF pose tracking. This involves a sequence of tools like SAM3, SAM3D, MoGe, HaWoR, and TAPIR.
2. Retargeting: The reconstructed human motion is mapped onto a robot's hand and arm. This process uses convex decomposition, MJCF scene generation, and sampling-based MPC in MuJoCo Warp to ensure the motion is feasible for the robot.
3. Deployment: The retargeted motion is converted into joint trajectories for specific hardware (dual-UR3e arms and Sharpa Wave hands) and streamed to the real robot for replay.

Who it’s for

This is for researchers and developers working in robotics, specifically those focusing on on-device learning from demonstration (LfD) learning from demonstration (LfD) and human-to-robot motion retargeting.

Highlights

• End-to-end pipeline: Covers everything from raw video input to real-world robot execution.
• Robot-specific deployment: Specifically tested and deployed on dual-UR3e arms with Sharpa Wave hands.
• Integration of multiple AI tools: Combines vision-based reconstruction reconstruction and physics-based retargeting.