TX-Leo/HumanEgo

HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos

What it solves

HumanEgo enables robots to learn complex tasks from very short durations (minutes) of human egocentric videos. It addresses the challenge of transferring human demonstrations captured from a first-person perspective to robot execution without requiring extensive robot-specific training data.

How it works

The system uses a pipeline that converts raw first-person video (e.g., from Project Aria glasses) into robot-executable policies. It employs vision foundation models—including SAM 2, Grounding DINO, CoTracker, and Orient-Anything V2—to preprocess data. The training process uses a flow-matching policy that consumes "Interaction-Centric Tokens" (6DoF representations of hands and objects) and embodiment-agnostic images (where the robot arm is inpainted out) to predict future end-effector trajectories for a dual-arm robot.

Who it’s for

Robotics researchers and developers who want to train robots using human demonstrations rather than manual robot programming or large-scale robot datasets.

Highlights

  • Zero-Shot Learning: Learns from minutes of human egocentric video.
  • Embodiment-Agnostic: Uses inpainted images and interaction tokens to decouple the human demonstration from the specific robot hardware.
  • Comprehensive Tooling: Includes a full pipeline for data collection (Project Aria), preprocessing, training, and real-robot inference.
  • Open Dataset: Provides released datasets and pretrained checkpoints for tasks like serving bread and watering flowers.

Related

  • Project
    GaTech-RL2/EgoVerse

    EgoVerse is a framework and dataset for robot learning from egocentric human demonstrations, providing tools for data processing, training, and evaluation.

  • Project
    Open-X-Humanoid/HEX

    A whole-body vision-language-action framework for humanoid robots that enables cross-embodiment manipulation by aligning heterogeneous robot states into shared body-part slots.

  • Project
    BeingBeyond/Being-H

    A family of human-centric embodied foundation models, including VLA and WAM architectures, designed to enable robots to learn and generalize actions from human videos.

  • Project
    facebookresearch/Ego4d

    A toolkit and dataset infrastructure for Ego4D and Ego-Exo4D, providing tools to download, process, and train models on massive first-person and multi-view video datasets.

  • Dispatch
    OpenAI One-Shot Imitation Learning

    OpenAI introduces a meta-learning framework for one-shot imitation learning that allows robots to learn new tasks from a single demonstration and generalize to new situations without task-specific engineering.