nvidia-isaac/video_to_data

Video to Data (V2D) – From human demo video to robot‑ready training data

What it is – An end‑to‑end, modular pipeline that turns ordinary video recordings of people performing tasks into simulation‑ready 3‑D assets and physics‑grounded reinforcement‑learning (RL) policies for robots. The repository is split into four main packages that can be run independently or chained together:

Package	Role	How it runs
video_ingestion_agent	Parses a raw video, segments it into action clips, builds an entity‑relation scene graph and stores per‑frame SigLIP‑2 embeddings. The output is a searchable action database (graph.db + vector.db).	Python virtual‑env + a vLLM server (LLM‑backed retrieval).
reconstruction	Converts the selected clips into per‑frame depth maps, object masks, textured meshes, 6‑DoF object poses, and SMPL human‑body parameters. Multi‑view pipelines handle stereo or rosbag inputs.	Docker containers (one per vision module).
robotic_grounding	Retargets the reconstructed human motion onto a robot model (e.g., Sharpa), then trains an RL policy in NVIDIA Isaac Lab using RSL‑RL PPO. Includes utilities for dataset conversion and policy evaluation.	Docker container built on top of Isaac Lab 2.3.1.
robotic_grounding/flash_chord	Optional accelerated training path that uses FlashSAC/FlashCHORD (Newton / Warp / JAX) for faster reference‑tracking RL.	Separate container with JAX‑based stack.

Why it matters – V2D bridges the gap between easily captured human demonstrations (a phone video) and the high‑fidelity simulation data required to train robot policies that can be deployed on real hardware. By exposing each stage as a file‑based, containerized module, researchers can:

• Re‑use intermediate results (e.g., depth maps) without re‑running the whole pipeline.
• Swap out or improve individual components (different depth estimator, different RL algorithm) while keeping the rest unchanged.
• Run the full workflow on a single RTX A6000 (or L40/L40S) GPU, making it accessible to many labs.

Key components & technologies

• LangGraph‑driven ingestion agent – LLM‑guided segmentation, verification, and natural‑language retrieval of action clips.
• Vision stack – Grounding‑DINO, SAM‑2, FoundationPose, SAM3D‑Body, MoGe, etc., packaged as Docker images.
• Isaac Lab 2.3.1 + RSL‑RL PPO – Simulation environment and RL algorithm for policy learning.
• FlashCHORD – Optional JAX‑based accelerated PPO/FlashSAC training.
• Typed data contracts – v2d_common defines explicit dataclasses (DepthImage, CameraIntrinsics, Transform3d, …) to keep module interfaces clean.

Typical workflow

1. Ingest a video with video_ingestion_agent/scripts/run_ingestion.py. The agent creates a searchable database of action clips.
2. Retrieve clips of interest via natural‑language query (run_retrieval.py) or the provided Gradio UI.
3. Reconstruct 3‑D data from the selected clips using the Docker‑based modules (run_mv_hoi_reconstruction for full multi‑view pipelines).
4. Ground the human motion onto a robot model and train a policy with Isaac Lab (robotic_grounding/scripts/run_pipeline_docker.py → train.py).
5. (Optional) Accelerate training with FlashCHORD.
6. Visualize the retargeted motions in a browser‑based gallery.

Getting started – The README provides a quick‑start sequence:

# 1. Video ingestion
cd video_ingestion_agent
uv venv .venv && source .venv/bin/activate
uv pip install -e ".[all]"
python scripts/serve.py -c configs/ingestion.yaml   # start vLLM server
python scripts/run_ingestion.py path/to/video.mp4 -c configs/ingestion.yaml --no-verify -o runs/my_run
python scripts/run_retrieval.py "find clips where someone picks up a mug" -d outputs/ -c configs/retrieval.yaml

# 2. Reconstruction (Docker)
cd reconstruction
./scripts/install_packages.sh
./scripts/build_containers.sh
python -m v2d.pipelines.run_mv_hoi_reconstruction \
  --rosbag_path /data/rosbags/session1 \
  --output_dir /data/datasets/session1 \
  --calibration_camera_params_path /data/datasets/calibration/extrinsics/edex \
  --obj_mesh_path /data/meshes/object.glb

# 3. Robotic grounding & RL training
cd robotic_grounding
bash workflow/setup_deps.sh && git lfs pull
python scripts/run_pipeline_docker.py --build-only
python scripts/run_pipeline_docker.py taco --hmd $HOME/datasets/human_motion_data --max-sequences 2
./workflow/run.sh build && ./workflow/run.sh start
python scripts/rsl_rl/train.py --task Sharpa-V2D-v0

The documentation links (E2E workflow guide, tech report, dataset collection on Hugging Face) give deeper details and reproducibility checks.

Who should use it – Researchers and engineers working on robot learning from demonstration, especially those interested in:

• Vision‑based 3‑D reconstruction of human‑hand interactions.
• End‑to‑end pipelines that go from raw video to deployable policies.
• NVIDIA‑centric simulation and RL stacks (Isaac Lab, RSL‑RL, FlashCHORD).

License & contribution – The repo follows the standard NVIDIA open‑source licensing (see each package’s LICENSE file). Contributions are welcomed; a contributing guide lives in the reconstruction README and outlines how to add new reconstruction modules (Docker image, weight‑download entry point, typed API).