OpenAI Safety Gym Release

OpenAI has released Safety Gym, a suite of environments and tools for measuring progress toward reinforcement learning (RL) agents that can respect safety constraints while training. This release provides a standardized method for comparing algorithms to determine how effectively they avoid costly mistakes during the learning process, which is critical for deploying RL in real-world robotics or internet-based tasks.

The Risk of Exploration in Reinforcement Learning

Reinforcement learning agents learn optimal behaviors through trial and error, a process known as exploration. However, exploration is fundamentally risky because agents may attempt dangerous behaviors that lead to unacceptable errors before they learn to avoid them. For example, an autonomous robot arm in a factory might flail randomly during early training, posing a safety risk to nearby human workers.

While simple physical barriers or emergency shut-offs can mitigate risks in restricted environments, general RL systems operating under diverse conditions require more robust algorithmic approaches to safe exploration.

Constrained Reinforcement Learning as a Formalism

To address the safe exploration problem, OpenAI has adopted the formalism of constrained reinforcement learning. Unlike standard RL, where an agent seeks only to maximize a reward function, constrained RL introduces cost functions that the agent must constrain.

Reward vs. Cost Functions

In standard RL, safety is often handled by incorporating a penalty into the reward function. This approach is problematic because reward design is difficult; if the reward for completing a task is high enough, an agent may accept frequent collisions or dangerous behaviors to maximize its total return.

In constrained RL, developers specify an acceptable cost rate (e.g., an acceptable collision rate) at the start of training. The algorithm then adjusts the penalty (the "cost fine") until the agent meets that specific safety requirement, allowing the system to prioritize outcomes over arbitrary trade-offs.

Safety Gym Environments and Tools

Safety Gym provides a diverse set of environments where robots must navigate cluttered spaces to achieve specific tasks. The suite includes three robot types, three main tasks, and two levels of difficulty for each task.

Robot Configurations

Point: A simple 2D robot with actuators for turning and moving forward/backward, featuring a small square for pushing tasks.
Car: A robot with two independently-driven parallel wheels and a free-rolling rear wheel, requiring coordinated actuator control for movement.
Doggo: A quadruped with bilateral symmetry and controls for hip azimuth, elevation, and knee angle.

Task Types

Goal: The robot must move to a series of goal positions.
Button: The robot must press a series of goal buttons.
Push: The robot the robot must move a box to a series of goal positions.

In these environments, running into clutter is defined as an unsafe action, triggering a red warning light and incurring a cost separate from the task reward.

Benchmarking and Preliminary Results

OpenAI evaluated several standard RL and constrained RL algorithms using the Safety Gym benchmark suite, including PPO, TRPO, Lagrangian penalized versions of PPO and TRPO, and Constrained Policy Optimization (CPO).

Preliminary results indicated that the environments range significantly in difficulty, with some being easy to solve and others too challenging for current techniques. Notably, OpenAI found that Lagrangian methods performed surprisingly better than CPO, which contradicts previous findings in the field.

To support reproducibility, OpenAI released the algorithms code via the Safety Starter Agents repository.

Future Research Directions

OpenAI identified three primary areas for further refinement of constrained RL:

Improving agent performance within the current Safety Gym environments.
Using Safety Gym to investigate distributional shift and safe transfer learning.
Integrating constrained RL with implicit specifications, such as human preferences, for defining rewards and costs.

OpenAI suggests that safety measurements could eventually be integrated into developer assessment schemes or used by government agencies to create safety standards for AI systems.