GaTech-RL2/EgoVerse

What it solves

EgoVerse provides a framework and dataset for robot learning from egocentric data. It addresses the challenge of training robots to perform tasks by leveraging human demonstrations recorded from a first-person perspective, allowing robots to learn from diverse, real-world human movements across different embodiments.

How it works

The project implements a pipeline for processing raw egocentric data (such as from Aria glasses or Aloha) into a standardized Zarr format. It uses a unified embodiment system where human demonstrations are collapsed into a single human embodiment category to simplify joint training with robots (like the Eva robot). The system includes training scripts powered by PyTorch Lightning and Hydra, supporting algorithms like EgoMimic (HPT-based) and Pi, and provides tools for visualizing data with a local web app that overlays recorded actions and keypoints onto video frames using camera intrinsics.

Who it’s for

It is designed for researchers and developers working on robot learning, imitation learning, and egocentric vision, specifically those looking to train robot policies using human-to-robot demonstration data.

Highlights

• Unified Embodiment: Collapses multiple human data sources into a single human embodiment to align human and robot orientations for joint training.
• Comprehensive Tooling: Includes data processing scripts, training pipelines, and a dedicated interactive dataset browser for visualizing trajectories and keypoints.
• Multi-source Data Support: Provides tools to process data from various sources including Aria, Aloha, and other external datasets.
• Scalable Training: Supports Distributed Data Parallel (DDP) training via PyTorch Lightning and Hydra.