unitreerobotics/xr_teleoperate

What it solves

This project provides a system for teleoperating Unitree humanoid robots using Extended Reality (XR) devices. It allows a human operator to control the robot's arms and dexterous hands in real-time, providing a first-person perspective and the ability to record teleoperation data for further use in imitation learning.

How it works

The system connects an XR headset (such as Apple Vision Pro, Meta Quest 3, or PICO 4 Ultra) to a host computer and the robot. It uses a web-based interface (Vuer) and WebRTC for low-latency image streaming from the robot's head camera to the operator. The operator's movements are tracked via hand or controller tracking in the XR device, which are then mapped to the robot's degrees of freedom (DoF) and sent as control commands via the unitree_sdk2_python library.

Who it’s for

Robot developers and researchers who own Unitree humanoid robots (like the G1, H1, H2, or R1) and want to perform precise remote control or collect high-quality demonstration data for AI training.

Highlights

• Broad Hardware Support: Compatible with multiple Unitree humanoid models (G1, H1, H2, R1) and various end-effectors including dexterous hands (Dex1-1, Dex3-1, BrainCo).
• XR Integration: Supports a variety of high-end XR headsets for immersive first-person control.
• Data Collection: Includes a built-in recording mode to save teleoperation episodes for imitation learning tasks.
• Flexible Control Modes: Offers both hand-tracking and controller-tracking input modes, as well as immersive, ego, and pass-through display modes.