apple-aiml-research/ml-egodex

What it solves

EgoDex provides a large-scale, high-quality dataset and benchmark to help AI models learn complex, dexterous hand manipulation. By providing paired egocentric video and precise 3D pose annotations, it addresses the lack of available data for training robots or AI to perform intricate tabletop tasks.

How it works

Collected using Apple Vision Pro and ARKit, the dataset consists of 829 hours of 1080p video across 194 different tasks. Each video is paired with an HDF5 file containing:

• 3D Pose Annotations: SE(3) transforms for the head, upper body, and 68 hand joints.
• Confidence Scores: Values indicating the reliability of the joint detection.
• Language Metadata: Descriptions of the tasks generated by LLMs and VLMs (GPT-4).

The repository includes Python scripts to load the data into PyTorch, visualize the 3D skeletal data, and compute distance metrics for evaluating trajectory prediction quality.

Who it’s for

• Robotics Researchers: Those developing robot foundation models or vision-language-action (VLA) models for dexterous manipulation.
• Computer Vision Engineers: Researchers working on 3D human pose estimation from egocentric video.
• AI Developers: Anyone training models to mimic human hand movements for robotic deployment.

Highlights

• Massive Scale: Over 800 hours of egocentric video with 3D annotations.
• Diverse Tasks: Covers 194 different active tabletop manipulation tasks.
• High Precision: Uses ARKit on visionOS for detailed skeletal tracking of hands and body.
• Integrated Benchmarking: Includes tools to evaluate the quality of predicted dexterous trajectories.