OpenAI RL-Teacher Release

OpenAI has released RL-Teacher, an open-source implementation designed to train AI agents using occasional human feedback. This approach replaces the need for hand-crafted reward functions, which is particularly useful for reinforcement learning problems where rewards are difficult to specify and serves as a step toward developing safe AI systems.

Core Components of RL-Teacher

RL-Teacher provides a modular system consisting of three primary components:

• Reward Predictor: A pluggable component that learns to predict which actions an agent takes would be approved of by a human.
• Example Agent: An agent that learns via a function specified by the reward predictor. The release includes three pre-integrated algorithms, including OpenAI Baselines PPO.
• Web-App: A user interface that allows humans to provide the feedback necessary to train the reward predictor.

Technical Implementation and Usage

The RL-Teacher system is lightweight, consisting of fewer than 1,000 lines of Python code, excluding the agents.

To launch an experiment, users can set up a web server and run the following command:

python rl_teacher/teach.py -p human --pretrain_labels 175 -e Reacher-v1 -n human-175

Humans provide feedback through a simple web interface that can be run locally or on a separate machine. Full documentation is available on the project's GitHub repository.