OpenAI Summer Fellows 2018 Final Projects

Overview

OpenAI's 2018 Summer Fellows program concluded with a series of research projects targeting critical bottlenecks in artificial intelligence, specifically in reinforcement learning (RL) generalization, the scalability of neural network training, and the expressivity of generative models.

Reinforcement Learning and Generalization

Research conducted by the 2018 fellows focused on quantifying and mitigating overfitting in RL agents to ensure they can transfer knowledge across different environments.

Quantifying Generalization via CoinRun

Karl Cobbe developed CoinRun, a procedurally generated game designed to measure how well RL agents generalize to new environments. The research revealed that agents often overfit to training sets, even when those sets are surprisingly large. To reduce this overfitting, Cobbe implemented deeper convolutional architectures and supervised learning techniques, including:

  • L2 regularization
  • Dropout
  • Data augmentation
  • Batch normalization

Pre-training for Sample Efficiency

Josh Meier investigated the RL transfer problem, where agents typically struggle to transfer knowledge between tasks. He discovered that pre-training neural networks to model the environment using a large generative model on unsupervised observations, followed by fine-tuning the model on-policy using Proximal Policy Optimization (PPO), leads to increased sample efficiency and improved transfer across tasks. This approach combined scaling transformer networks, fine-tuning language models, and PPO.

Analysis of Overfitting in RL

Xingyou (Richard) Song analyzed the types of overfitting that occur in reinforcement learning from both optimization and synthetic perspectives. His work focused on observation overfitting and the factors within the optimization landscape that contribute to generalization gaps. Additionally, Song collaborated with Joshua Meier to evaluate state-of-the-art methods on the Sonic The Hedgehog environment, analyzing why certain generative modeling and special architectures fail on large datasets.

Distributed Training and the Science of AI

Sam McCandlish studied the use of large-scale computing hardware for distributed neural network training. His research identified predictable patterns across a wide variety of machine learning tasks, ranging from MNIST to Dota. Specifically, he found that the tradeoff between the amount of experience and the training time required to achieve a a specific score is predictable.

Generative Models and Physics Priors

Research in generative modeling focused on improving sample quality, stability, and the ability to benchmark model expressivity.

Normalizing Flow Models

Johannes Otterbach studied Normalizing Flow models, which approximate data distributions through continuous deformation of a simpler distribution. He created artificial datasets that are intrinsically difficult for these models to approximate, providing a benchmark for future generative models to measure their flexibility and expressivity.

Energy-Based Models and Dynamics Knowledge

An unnamed fellow (previously an MIT undergrad) explored two primary areas of generative modeling and RL:

  1. Dynamics Knowledge Integration: The fellow developed a new architecture for better long-term physics predictions and explored ways to transfer dynamics information learned from videos and previous environments into new ones.
  2. Energy-Based Models (EBMs): The fellow worked on scaling and stabilizing the training of EBMs. By implementing a replay buffer, the fellow found that EBMs could generate higher-quality samples than other state-of-the-art likelihood models, demonstrating strong out-of-distribution generalization, compositional ability, and the ability to inpaint and restore CIFAR-10 samples.

Sources