OpenAI Requests for Research 2.0

OpenAI has released Requests for Research 2.0, a set of seven unsolved problems derived from their internal research. This initiative aims to engage the broader research community by providing specific, high-impact technical challenges in reinforcement learning (RL), generative models, and automated reasoning.

Reinforcement Learning Challenges

OpenAI identifies several challenges focused on improving the stability and efficiency of reinforcement learning agents.

Multiplayer Game Environments (Slitherin’)

Researchers are tasked with implementing a multiplayer clone of the classic Snake game as a Gym environment. The goal is to develop an agent using self-play and an RL algorithm to solve the environment. Key technical hurdles include:

  • Environment Design: Creating a large field with multiple snakes that grow by eating fruit and die upon collision with walls, other snakes, or themselves.
  • Self-Play Instability: Overcoming the instability often associated with self-play (similar to GAN instability) by experimenting with training policies against a distribution of past policies.
  • Behavioral Analysis: Determining if agents learn complex strategies such as attacking, trapping, or ganging up on competitors.

Parameter Averaging in Distributed RL

This request focuses on the sample complexity and communication overhead of distributed RL algorithms. OpenAI suggests exploring parameter averaging schemes to reduce communication bandwidth by updating workers independently and averaging parameters infrequently. Potential benefits include:

  • Improved Exploration: Agents with different parameters may exhibit better exploration behavior.
  • Sample Complexity: Investigating how different averaging schemes, including algorithms like EASGD, affect the overall learning efficiency.

Generative Models and Architecture

OpenAI proposes research into the efficiency and transferability of Transformer-based models.

Transfer Learning via Generative Models

This challenge explores whether pre-training on a diverse set of tasks can improve performance on a new task. The proposed methodology involves:

  • Training policies for 11 Atari games.
  • Fitting a generative model (such as a Transformer) to trajectories from 10 of those games.
  • Fine-tuning the model on the 11th game to quantify the benefit of pre-training.
  • Analyzing how the model size and the amount of available data for the 11th game affect the transfer learning benefit.

Transformers with Linear Attention

OpenAI seeks a way to replace the softmax attention mechanism in Transformers with linear attention. This would allow the Transformer to be converted into an RNN with fast weights, making RL rollouts over large contexts feasible. The specific goal is to achieve the same bits per character/word on a language modeling task using a linear-attention transformer without significantly increasing the total parameter count. OpenAI notes that this may require higher dimensional key/value vectors.

Automated Reasoning

OpenAI is challenging the community to develop automated solutions for Olympiad inequality problems. These problems are characterized by being simple to express but requiring clever manipulations to solve. The objective is to build a dataset of these problems and create a program capable of solving a large fraction of them, potentially using a learned policy to reduce the branching factor during the search for a solution.

Starter Problems (Warmups)

To help researchers get started, OpenAI provided two solved starter problems:

  • LSTM XOR Problem: Training an LSTM to determine the parity of a binary sequence, testing the performance difference between fixed-length and variable-length binary strings.
  • Snake Gym Environment: Implementing a basic Snake game as a Gym environment and solving it with a reinforcement learning algorithm.

Sources