GaTech-RL2/EgoVerse

What it solves

EgoVerse provides a framework and dataset for robot learning from egocentric (first-person) data. It specifically addresses the challenge of training robots to perform tasks by leveraging human demonstrations recorded from a first-person perspective, allowing robots to learn from diverse, real-world human data across different embodiments.

How it works

The project implements a pipeline for processing raw egocentric data (such as from Aria glasses or Aloha) into a standardized Zarr format. It uses a unified embodiment system where human demonstrations and robot data (specifically for the Eva robot) are aligned so that their end-effector orientations and poses are consistent. This allows for joint training and visualization using algorithms like ACT, Pi, and EgoMimic (HPT-based).

Who it’s for

Researchers and developers in robot learning and embodied AI who want to train robots using human egocentric demonstrations and standardized data processing tools.

Highlights

• Multi-Embodiment Support: Collapses various human data sources into a unified human embodiment for easier joint training with robots.
• Standardized Data Format: Uses Zarr stores for efficient data handling and includes mandatory camera intrinsics for accurate rendering.
• Integrated Visualization: Includes a local web app (latent_inspector.py) for the interactive browsing of episodes, including overlays for trajectories, orientations, and keypoints.
• Diverse Algorithm Support: Provides training scripts for multiple imitation learning algorithms including ACT and Pi.