NVIDIA/IsaacCapture

The unified framework for sim & real robot teleoperation

What it solves

It addresses the data bottleneck in robot learning by providing a unified framework for collecting high-fidelity human demonstration data, streamlining how devices are integrated and ensuring data interoperability across different robots.

How it works

Isaac Teleop provides a standardized device interface and a flexible retargeting framework that works across both simulated and real-world teleoperation. It allows users to capture egocentric and robot data using various hardware, such as XR headsets and gloves, to control robots like gripper hands or full-body loco-manipulators.

Who it’s for

Robotics researchers and developers who need to collect high-quality training data for robot learning through human teleoperation in both simulation and reality.

Highlights

  • Unified stack for both simulation and real-world teleoperation.
  • Standardized device interfaces to simplify hardware integration.
  • Flexible retargeting framework for mapping human movements to robot actions.
  • Support for diverse use cases including XR-based manipulation and full-body loco-manipulation.

Related

  • Project
    droidrun/mobile-harness

    Mobile Harness is a Python‑based, Markdown‑driven library that lets LLM agents control Android and iOS devices (locally via ADB/Portal or remotely via Mobilerun cloud). It offers a unified API for UI interaction, screenshots, app launch, and accessibility‑tree queries, and includes benchmarks showing near‑perfect performance on AndroidWorld and iOSWorld task suites.

  • Project
    cybergalactic/PythonVehicleSimulator

    A Python-based simulation environment for marine vehicles that provides models for AUVs, USVs, and ships to test guidance, navigation, and control systems.

  • Project
    Qpu523/VPSCI-Dataset

    VPSCI is a large, high‑resolution dataset of vehicle‑pedestrian near‑miss interactions generated by a three‑stage reinforcement‑learning pipeline in the CARLA simulator. It contains ~198 k episodes across eight intersection scenarios, with CSV files logged at 20 Hz and supporting videos. Human studies show the generated interactions are indistinguishable from real‑world footage, making the dataset valuable for autonomous‑driving safety research.

  • Project
    kesai-labs/py123d

    py123d is a Python library that unifies many autonomous‑driving datasets (Argoverse 2, nuScenes, Waymo, nuPlan, etc.) by converting them to a fast, zero‑copy Apache Arrow format and exposing a single API for cameras, lidar, maps, and labels. It includes CLI tools for downloading and converting data, Hydra‑based pipelines, and an interactive 3‑D viewer. Install via `pip install py123d` (with optional extras for each dataset) and use the provided commands to fetch, convert, and visualise data.