NVlabs/GRAIL

What it solves

GRAIL addresses the difficulty of obtaining high-quality, metric 4D training data for humanoid robots performing complex tasks that combine movement (locomotion) and object interaction (manipulation). Instead of relying on expensive real-world data collection, it provides a fully digital pipeline to synthesize this data.

How it works

GRAIL uses a multi-stage pipeline to generate synthetic training data:

1. 3D Asset Generation: Creates procedural terrain (like stairs) or AI-generated 3D objects.
2. 2D HOI Generation: Combines 3D assets, simulator-ready scenes, and video foundation models (such as Kling or MiniMax-H3) to create 2D human-object interaction (HOI) videos.
3. 4D HOI Reconstruction: Converts these 2D videos into metric 4D trajectories.
4. Retargeting: Maps these trajectories to a specific robot, such as the Unitree G1 humanoid.
5. Policy Training: Trains task-general policies for activities like picking up objects, sitting, and traversing terrain.

Who it’s for

Robotics researchers and developers working on humanoid loco-manipulation and sim-to-real transfer, specifically those looking to generate large-scale synthetic datasets for training visual policies.

Highlights

• Full Digital Pipeline: Synthesizes 4D trajectories from 3D assets and video priors.
• Sim-to-Real Transfer: Policies trained solely on GRAIL data have successfully transferred to real-world object pick-up and stair-climbing.
• Video Foundation Model Integration: Leverages external video models to generate realistic interaction priors.
• Task-General Capabilities: Supports a wide range of motions including tabletop and ground pickup, sitting, and slope/stair traversal.