real-stanford/umi-on-legs

UMI on Legs: Making Manipulation Policies Mobile with Manipulation-Centric Whole-body Controllers

What it solves

UMI on Legs provides a way to make manipulation policies mobile by integrating them with whole-body controllers (WBC) on quadruped robots equipped with arms. It allows users to plug-and-play existing visuomotor policies onto a robot dog, enabling the robot to perform manipulation tasks while maintaining balance and stability.

How it works

The framework combines real-world human demonstrations (via the Universal Manipulation Interface) with whole-body controllers trained in simulation. The system consists of several components:

  • Simulation Training: Whole-body controllers are trained using reinforcement learning (PPO) in simulation environments (IsaacGym).
  • Real-World Deployment: The trained controllers are deployed to real robots, utilizing an iPhone odometry application for positioning and an ARX5 SDK for arm control.
  • UMI Integration: It leverages the Universal Manipulation Interface to translate human demonstrations into robot-executable manipulation trajectories.

Who it’s for

Researchers and developers working on embodied AI, robotics, and specifically those looking to integrate manipulation skills with mobile quadruped platforms.

Highlights

  • Plug-and-Play Mobility: Enables the transition of existing visuomotor policies to mobile quadruped robots.
  • Simulation-to-Real Transfer: Includes tools for robustifying sim2real transfer of the whole-body controllers.
  • Modular Design: The codebase is organized as separate submodules (WBC simulation training, real-world deployment, iPhone odometry, and ARX5 SDK) for easy integration into other systems.
  • Comprehensive Hardware Guide: Provides detailed assembly, 3D printing, and network setup guides for Unitree robots.

Related

  • Project
    NVlabs/ProtoMotions

    A GPU-accelerated simulation and learning framework for training physically simulated digital humans and humanoid robots, supporting large-scale motion learning and sim-to-real deployment.

  • Project
    OpenMind/OM1

    A modular AI runtime for creating and deploying multimodal agents across digital environments and physical robots, supporting diverse sensors and AI models.

  • Project
    Open-X-Humanoid/HEX

    A whole-body vision-language-action framework for humanoid robots that enables cross-embodiment skill transfer and long-horizon manipulation.

  • Project
    manumerous/wb_humanoid_mpc

    A real-time Nonlinear Model Predictive Controller (NMPC) for humanoid robots that optimizes full-order torque-level dynamics for loco-manipulation.

  • Project
    facebookresearch/project_superdex

    Project SuperDex is an open‑source simulation platform for dexterous robot manipulation. It bundles a contact‑first physics engine, a robotics SDK, a visual authoring GUI (Studio), and a Gymnasium‑style RL interface (Lab). Users can create robot/scene assets, run high‑fidelity simulations, and train reinforcement‑learning or MPC policies via Python bindings. Future modules add Unreal‑Engine‑based VR teleoperation for Quest 3. The core is Apache 2.0‑licensed; Python wheels are installable with `uv pip install superdex`.