OpenAI Benchmarking Safe Exploration in Deep Reinforcement Learning
TL;DR
OpenAI has introduced the Safety Gym benchmark suite to standardize constrained reinforcement learning (RL) as the primary formalism for safe exploration. This allows researchers to measure progress in developing RL agents that can learn optimal policies through trial and error without violating critical safety constraints in high-dimensional continuous control environments.
Standardizing Constrained RL for Safe Exploration
Safe exploration is critical for RL agents that interact with the real world, particularly in robotics systems that interact with humans, where certain errors are unacceptable. While most RL agents are currently trained in simulation, OpenAI posits that the shift toward training agents directly in the real world will necessitate a more robust approach to safety.
OpenAI proposes that constrained RL be used as the main formalism for safe exploration. Constrained RL allows agents to learn optimal policies while adhering to a specific set of safety constraints, ensuring that the agent does not engage in behaviors that are unacceptable during the exploration phase of learning.
The Safety Gym Benchmark Suite
To measure research progress on constrained RL, OpenAI has released the Safety Gym benchmark suite. Safety Gym provides a new slate of high-dimensional continuous control environments designed specifically to test an agent's ability to maintain safety constraints while achieving its goal.
These environments are simulated environments that provide a a high-dimensional continuous control problem where the göreve is the same as typical RL environments—maximizing reward—but with aset of constraints that the same agent must follow.
Establishing Baselines for Future Research
OpenAI has benchmarked several constrained deep RL algorithms on the Safety Gym environments to establish baselines. By providing these benchmarks, OpenAI provides a baseline for future research in the same field of safe exploration and safe reinforcement learning.