NVlabs/GRAIL

What it solves

GRAIL addresses the difficulty of obtaining large-scale, high-quality training data for humanoid robots performing complex tasks that involve both movement (locomotion) and object interaction (manipulation). Instead of relying on expensive real-world data collection, it provides a fully digital pipeline to synthesize realistic training trajectories.

How it works

The pipeline generates data through several stages:

1. Asset Generation: It creates 3D assets, including procedural terrains (like stairs) and AI-generated objects.
2. 2D-HOI Generation: It uses Blender and video foundation models (such as Kling or MiniMax-H3) to create 2D human-object interaction (HOI) videos.
3. 4D-HOI Reconstruction: These videos are reconstructed into metric 4D trajectories.
4. Retargeting: The reconstructed human motions are retargeted to a specific robot, such as the Unitree G1.
5. Policy Training: The resulting data is used to train task-general policies for actions like picking up objects, sitting, and traversing uneven terrain.

Who it’s for

It is designed for robotics researchers and engineers working on humanoid loco-manipulation and sim-to-real transfer.

Highlights

• End-to-End Digital Pipeline: Generates data from 3D assets and video priors without needing real-world human demonstrations.
• Sim-to-Real Transfer: Policies trained solely on GRAIL-generated data have successfully transferred to real-world object pick-up and stair-climbing.
• Versatile Task Support: Supports a wide range of behaviors including tabletop and ground pickup, whole-body manipulation, and terrain traversal (slopes, curbs, and stairs).
• Integration with Video Models: Leverages state-of-the-art video foundation models to synthesize realistic interaction priors.