OpenAI Scholars 2021: Final Projects

OpenAI has released the final projects from the 2021 Scholars program, showcasing a range of research initiatives focused on scaling laws, model architecture, and the alignment of AI systems. The program aims to provide research experience to individuals from underrepresented groups, enabling them to contribute to the frontier of deep learning.

Scaling Laws and Language Transfer

Christina Kim's research explores the efficacy of English pre-training for transfer learning across different languages. The study found that pre-trained English models provide the most benefit when learning German, followed by Spanish, and then Chinese. Furthermore, the transfer from English to these three languages scales predictably in terms of compute, data, and parameters.

Reward Modeling and Feedback Loops

Research into how AI models interact with human values and data ecosystems was a primary focus for several scholars:

• Large Scale Reward Modeling: Jonathan Ward demonstrated that reward models, which are typically trained on human feedback, can be trained using large-scale structured feedback extracted from websites.
• Opinion Modeling: Danielle Ensign investigated the impacts of deep learning models on the ecosystem of opinions and the effects of iterative training on outputs from previous models (feedback loops).

Model Architecture and Representation Learning

Several projects focused on the technical constraints and performance of various AI architectures:

• Contrastive Learning: Ellie Kitanidis pre-trained a language representation model using a purely contrastive objective to study its generalizability and scalability compared to traditional language modeling objectives. Legg Yeung used the "SET" card game to investigate the relationship between vector representation dimension and task composition, finding that non-contrastive models with X parameters can solve games that contrastive models with 2X+ parameters cannot.
• Transformer Variants: Shola Oyedele studied scaling laws for different transformer architecture variants to understand the impact of architecture on the tradeoff between training cost and model performance.
• Tokenization: Sam Gbafa explored the tradeoffs and scaling of different language tokenization schemes, including an approach to learn a sequence's segmentation rather than using a predefined one.

Reinforcement Learning and Compute Efficiency

Projects in this control and compute domains explored how to model complex behaviors and optimize resource use:

• Continuous Control: Florentine (Tyna) Eloundou developed the "multiple experts, multiple objectives" (MEMO) approach. This method allows an agent to learn a single conditional policy from examples of success from multiple experts with different objectives, which a supervisor can then orient at their discretion.
• Test Time Compute: Kudzo Ahegbebu explored the generalizability of models leveraging test time compute across autoregressive transformers, deep equilibrium models, and graph neural networks. The research investigated whether small adaptive models can use test time compute to overcome the limitations of a smaller number of learnable parameters.

Scholar Insights on Deep Learning Research

The 2021 Scholars shared several key pieces of advice for those entering the field of AI research:

"My advice to someone starting in deep learning research is to take your time to understand insights from fundamental papers and remember that the field is still relatively new. There’s a lot of room for individuals to have an outsized impact."

"If you can, take a few months to carefully work through the 2019 fast.ai course (parts 1 and 2), Andrew Ng’s deep learning course on Coursera, David Silver’s RL Course, and Spinning Up in Deep RL... try to implement a few papers from scratch in pytorch."

"Find the simplest interesting idea that you can think of and build it!"

"I would recommend slowly stepping through clean open source implementations of well-known algorithms while reading their theoretical grounding."