OpenAI Robots That Learn: One-Shot Imitation Learning from Simulation

OpenAI has created a robotics system that can learn a new task after seeing it performed once. The system is trained entirely in simulation and then deployed on a physical robot, eliminating the need for real-world image training.

One-Shot Imitation Learning and VR Integration

OpenAI's new algorithm, one-shot imitation learning, allows humans to communicate tasks to a robot by performing them in Virtual Reality (VR). After a single demonstration in the simulator, the robot can solve the same task from an arbitrary starting configuration in the physical world.

System Architecture: Vision and Imitation Networks

The system is powered by two distinct neural networks that handle perception and task execution:

Vision Network

The vision network converts images from the robot's camera into state representations of object positions. To ensure the system generalizes to the physical world, the network is trained using domain randomization—exposing it to hundreds of thousands of simulated images with varying lighting, textures, and object perturbations. The vision system is never trained on real images.

Imitation Network

The imitation network infers the intent of a task from a demonstration and generalizes that intent to new settings. It is trained on dozens of tasks with thousands of demonstrations each, using pairs of trajectories that perform the same task from different starting states. Through supervised learning, the network predicts the action a demonstrator took at a specific observation, learning to prioritize the relevant portions of the task over the specific starting locations of objects.

Application to Block Stacking

OpenAI demonstrated the system's capabilities using a block stacking task where the robot creates color-coded stacks of blocks.

Training with Scripted Policies

Because block stacking is a sufficiently simple task, OpenAI used a scripted policy in simulation to generate the training data for the imitation network. Despite being trained on scripted data, the imitation network successfully parsed demonstrations produced by humans at test time, even though it had not previously encountered "messy" human data.

Scaling and Generalization via Soft Attention

The imitation network utilizes soft attention over the demonstration trajectory and the state vector of block locations. This architecture allows the system to:

• Handle demonstrations of variable length.
• Imitate trajectories longer than any seen during training.
• Stack blocks into configurations containing more blocks than any demonstration in its training set.

Robustness through Noise Injection

To ensure the policy is robust, OpenAI injected a modest amount of noise into the outputs of the scripted policy during training. This forced the scripted policy to demonstrate recovery behaviors when errors occurred, which in turn taught the imitation network how to handle disturbances from an imperfect policy. Without this noise injection, the imitation network typically failed to complete the stacking task.