Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model – OpenAI 2016

TL;DR

OpenAI introduced a technique for transferring control policies learned in simulation to real‑world robots. At each time step the method uses the simulation policy to predict the next state, then queries a learned deep inverse dynamics model to select the real‑world action that would produce that state, thereby bypassing the need for accurate dynamic parameters.

Method Overview

The approach first runs a control policy that was trained in simulation. Instead of executing the policy’s raw control signal on the real robot, it computes the state that the simulation expects after applying that signal. A deep inverse dynamics model, trained on real‑world data, then determines which real‑world action is most likely to achieve that expected next state. This inverse model replaces the missing or inaccurate dynamics terms (e.g., friction, contact, mass, geometry) that cause the sim‑to‑real gap.

Data Collection for the Inverse Dynamics Model

To train the deep inverse dynamics model, the authors propose an incremental data‑collection procedure. The robot executes exploratory actions, records the resulting state transitions, and uses these pairs (action, resulting state) to supervise the model. As more data are gathered, the model’s predictions improve, allowing the transfer method to refine its action selection over time.

Experimental Comparison

The authors evaluated their method against several baselines designed to handle simulation‑to‑real‑world discrepancies, including output error control and Gaussian dynamics adaptation. Experiments showed that the deep inverse dynamics approach compares favorably to these baselines, indicating that learning an inverse model from real data can effectively close the gap without explicit system identification.

Implications

By decoupling policy learning from accurate dynamics modeling, the technique makes it safer and more practical to develop complex robot behaviors in simulation before deploying them on hardware. It also highlights a general strategy: learn an inverse dynamics model from real interactions and use it to translate simulated plans into real‑world actions, a principle that can be applied to various robotic platforms and learning algorithms.