OpenAI Solving Rubik's Cube with a Robot Hand

OpenAI has developed a system capable of solving a Rubik's Cube using a human-like robotic hand. By training neural networks entirely in simulation and employing a new technique called Automatic Domain Randomization (ADR), the system can transfer complex dexterous manipulation skills to physical hardware, demonstrating that reinforcement learning can solve real-world problems requiring high degrees of freedom.

Automatic Domain Randomization (ADR)

Automatic Domain Randomization (ADR) enables the transfer of neural networks from simulation to the real world by automatically generating progressively more difficult simulated environments. This approach removes the need for a perfectly accurate physical model of the real world, which is often impossible to achieve for complex objects like robotic hands and Rubik's Cubes.

How ADR Works

ADR functions by starting with a single, non-randomized environment where a neural network learns to solve the cube. Once the network reaches a specific performance threshold, the system automatically increases the amount of domain randomization. This forces the network to generalize to a wider variety of conditions, and the process repeats as the network improves.

Randomized parameters include:

  • The size of the Rubik's Cube
  • The mass of the cube
  • The friction of the robot fingers
  • The visual surface materials of the hand

Advantages Over Manual Randomization

Unlike manual domain randomization, which requires human experts to specify ranges that are neither too narrow (hindering transfer) nor too broad (making learning impossible), ADR expands these ranges automatically without human intervention. In tests comparing ADR to manual randomization on a block-flipping task, ADR eventually doubled the transfer performance over the baseline without requiring human tuning.

Robustness and Emergent Meta-Learning

The system is highly robust to perturbations it never encountered during training, such as being prodded by a stuffed giraffe. This robustness is attributed to emergent meta-learning, where the network learns a general learning algorithm to rapidly adapt its behavior to the environment it is deployed in.

Testing for Adaptation

OpenAI tested this hypothesis by measuring the "time to success" for cube flips in simulation. When perturbations—such as resetting the network's memory, resetting dynamics, or breaking a joint—were applied, the time to success spiked initially but then decreased back to the baseline as the network relearned and adapted to the new conditions.

Neural Network Interpretability

By using non-negative matrix factorization from the interpretability toolbox, OpenAI visualized the network's memory. The high-dimensional vector was condensed into six color-coded groups. Analysis showed that each memory group corresponds to a semantically meaningful behavior, allowing researchers to identify if the network is about to spin the cube or rotate a face clockwise before the action actually occurs.

Performance and Challenges

While the system can solve the Rubik's Cube from any initial condition, it currently faces challenges with success rates depending on the complexity of the scramble:

  • Maximally difficult scrambles (26 face rotations): 20% success rate.
  • Simpler scrambles (15 rotations): 60% success rate.

Failures typically occur during the first few face rotations and flips, as the network must simultaneously solve the cube and adapt to the physical properties of the real world. If the cube is dropped, the network is capable of continuing the solve once the cube is placed back into the hand.

Development Path and Prototypes

To make the problem tractable, OpenAI utilized a series of custom cube prototypes as stepping stones before moving to a regular Rubik's Cube:

Prototype Position + Orientation Internal degrees of freedom (sensor)
Locked cub Vision 0 No sensor
Face cub PhaseSpace 2 PhaseSpace
Full cube PhaseSpace 6 PhaseSpace
Giiker cube Vision 6 Built-in sensors
Regular Rubik's Cube Vision 6 Vision

To break rotational symmetry, a small piece of each center cublet's sticker was cut out on the regular Rubik's Cube.

Sources