OpenAI Roboschool Release

OpenAI has released Roboschool, a set of physics-based environments designed for reinforcement learning. By porting existing MuJoCo tasks to the Bullet physics engine and introducing new complex scenarios, Roboschool provides a more accessible and robust framework for training agents to handle interactive control and multi-agent competition.

Physics Environments and MuJoCo Porting

Roboschool includes twelve initial environments. These include tasks previously available in MuJoCo, which have been ported to the Bullet physics engine and modified to be more realistic. In addition to the ported tasks, Roboschool introduces new challenges, including advanced versions of the Humanoid walker task and a multi-player Pong environment.

To facilitate immediate use, OpenAI provides trained policies for all environments in the agent_zoo folder of the GitHub repository, as well as a demo_race script that allows users to simulate a race between three robots.

Interactive Control and Robust Locomotion

Roboschool introduces environments that move beyond simple forward-motion goals to require interactive control. While previous OpenAI Gym environments often resulted in fragile policies where agents learned a single cyclic trajectory and failed when pushed, Roboschool's new 3D humanoid environments require agents to navigate toward a flag whose position varies randomly over time.

HumanoidFlagrun

This environment is designed to teach robots how to slow down and turn to reach a randomly moving target.

HumanoidFlagrunHarder

This environment increases the difficulty by allowing the robot to fall and giving it the time to recover. Episodes may start with the robot either upright or lying on the ground, and the agent is constantly bombarded by white cubes to force it off its trajectory.

OpenAI notes that the policies for these environments are multilayer perceptrons with no internal state. Because of this, the researchers believe that in some cases, the agents use their arms to store information to maintain stability and navigation.

Multi-Agent Training and Adversarial Dynamics

Roboschool supports the simultaneous training and execution of multiple agents within the same environment, starting with the RoboschoolPong environment.

Multiplayer training allows for several configurations:

• Training a single agent against itself.
• Training two different agents using the same algorithm.
• Training two different algorithms against each other.

OpenAI identifies a specific challenge in adversarial training: policy oscillation. In a policy gradient method applied to Pong, researchers observed that agents would overfit to their opponent's current strategy rather than learning a general solution. For example, one agent might learn to defend the top of the screen while the other learns to send the ball to the bottom. Once the first agent adapts to the bottom, the second agent adapts again, leading to a oscillating learning curve where neither agent learns a useful general strategy after hours of training. OpenAI suggests that while this adversarial interplay is tricky, it can provide a natural curriculum for developing sophisticated strategies.