nvidia-isaac/video_to_data

What it solves

Video to Data (V2D) solves the difficulty of creating high-quality, physics-grounded training data for robots by automating the conversion of raw human demonstration videos into simulation-ready assets and robot policies. It eliminates the need for manual data labeling or complex motion capture setups by extracting 3D scene information and human motion directly from video.

How it works

The pipeline consists of three composable stages:

1. Video Ingestion Agent: Uses a LangGraph-driven agentic workflow to segment videos into action clips, extract entity-relation scene graphs, and create a queryable database of frame embeddings (using SigLIP-2). This allows users to retrieve specific action clips using natural language.
2. Reconstruction: Employs containerized vision modules to transform RGB/stereo clips into 3D data, including depth maps, object masks, textured meshes, 6-DoF object poses, and SMPL human body parameters.
3. Robotic Grounding: Retargets the extracted human motion onto a specific robot embodiment (e.g., Sharpa) and uses NVIDIA Isaac Lab with RSL-RL PPO to train deployable policies based on the reconstructed scene and motion.

Who it’s for

Robotics researchers and engineers who need to generate large-scale, physics-accurate training datasets from human videos to train robot policies for real-world deployment.

Highlights

• Agentic Retrieval: Natural-language querying of demonstration videos to select specific clips for processing.
• Containerized Architecture: ML dependencies are isolated in Docker images, with a thin Python host orchestration layer.
• End-to-End Workflow: Covers the entire path from raw video ingestion to 3D reconstruction and robotic retargeting and RL training.
• Typed Data Contracts: Uses strongly-typed dataclasses for communication between modules to ensure data consistency.