OpenAI Ingredients for Robotics Research

OpenAI has released eight simulated robotics environments and a Baselines implementation of Hindsight Experience Replay (HER). These tools are designed to facilitate the training of models that can be transferred to physical robots, specifically targeting manipulation tasks that are more challenging than existing MuJoCo continuous control environments.

Simulated Robotics Environments

OpenAI provides eight new environments for Gym using the MuJoCo physics simulator, divided between two robot platforms:

Fetch Research Platform

  • FetchReach-v0: The robot must move its end-effector to a specific goal position.
  • FetchSlide-v0: The robot must hit a puck across a table so it slides and rests on a desired goal.
  • FetchPush-v0: The robot must push a box until it reaches a desired goal position.
  • FetchPickAndPlace-v0: The robot must pick up a box from a table and move it to a goal position above the table.

ShadowHand Robot

  • HandReach-v0: The robot must use its thumb and a selected finger to meet at a goal position above the palm.
  • HandManipulateBlock-v0: The robot must manipulate a block to achieve a desired position and rotation.
  • HandManipulateEgg-v0: The robot must manipulate an egg to achieve a desired position and rotation.
  • HandManipulatePen-v0: The robot must manipulate a pen to achieve a desired position and rotation.

Goal-Based Learning and Sparse Rewards

All new environments introduce a "goal" concept, such as a target position or orientation. By default, these environments use sparse rewards: a value of 0 if the goal is achieved (within tolerance) and -1 if it is not. While dense reward variants are available, OpenAI encourages the use of sparse rewards as they are more realistic for actual robotics applications.

Hindsight Experience Replay (HER)

To address the difficulty of learning from sparse rewards—where an agent may receive a constant -1 signal for many attempts—OpenAI released a Baselines implementation of Hindsight Experience Replay (HER).

HER allows an agent to learn from failure by substituting the original goal with the goal that was actually achieved during an episode. By pretending the achieved state was the intended goal, the agent receives a positive learning signal even when it fails the primary task. Over time, this enables the agent to learn how to achieve arbitrary goals, including the original target.

HER is an off-policy reinforcement learning algorithm that can be combined with other off-policy methods, such as Deep Deterministic Policy Gradient (DDPG). In testing, the combination of DDPG + HER with sparse rewards significantly outperformed vanilla DDPG and even DDPG + HER with dense rewards on challenging tasks like HandManipulateBlockRotateXYZ-v0.

Future Research Directions for HER

OpenAI identified several areas for improving HER and reinforcement learning in general:

  • Automatic Goal Creation: Learning a strategy for selecting hindsight goals rather than using hard-coded methods.
  • Unbiased HER: Utilizing importance sampling to derive an unbiased version of HER to prevent potential instabilities caused by goal substitution.
  • HER + HRL: Combining HER with Hierarchical Reinforcement Learning (HRL) to apply hindsight to actions generated by higher-level policies.
  • Richer Value Functions: Conditioning value functions on additional inputs like success thresholds or discount factors.
  • Faster Information Propagation: Investigating alternatives to target networks to speed up training without sacrificing stability.
  • HER + Multi-step Returns: Finding ways to use multi-step returns despite the off-policy nature of goal substitution.
  • On-policy HER: Exploring the combination of HER with stable on-policy algorithms like PPO via importance sampling.
  • Sample Efficiency: Designing algorithms that maintain performance even as the frequency of taking actions increases toward infinity.
  • Integration: Combining HER with advances such as Prioritized Experience Replay, distributional RL, or entropy-regularized RL.

Gym API Changes for Goal-Based Environments

To support these environments, the Gym API was updated with the following changes:

  • Observation Space: Goal-based environments now use a gym.spaces.Dict observation space containing the desired_goal, the achieved_goal, and the observation (the robot's state).
  • Reward Function Exposure: The environment's reward function is now exposed, allowing algorithms to re-compute rewards based on substituted goals.

Sources