OpenAI Scholars 2019 Final Projects
OpenAI has showcased the final projects of the 2019 OpenAI Scholars program, demonstrating how individuals from diverse academic and professional backgrounds applied deep learning and reinforcement learning to solve complex problems. These projects span natural language processing, robotic manipulation, medical treatment optimization, and model interpretability.
Natural Language Processing and Understanding
Several scholars focused on the application and optimization of transformer-based language models for specific tasks.
Fine-Tuning GPT-2 for Question Answering
Fatma Tarlaci experimented with fine-tuning the GPT-2 small model for Question Answering (QA) tasks. The project aimed to analyze the model's reasoning mechanisms, as QA requires a combination of language processing and reasoning skills.
Knowledge Distillation for Transformers
Edgar Barraza explored knowledge distillation to make powerful transformer language models more accessible for mobile devices. The project utilized a large, well-trained transformer as a "teacher" to train a smaller, untrained "student" network, reducing the model size without sacrificing significant capability.
Sentiment Analysis via Reinforcement Learning
Helen (Mengxin) Ji proposed models that combine reinforcement learning (RL) and supervised NLP methods to predict sentence sentiment. The results indicated that adding RL methods can improve performance over transformer models and produce results comparable to the-pre trained BERT model, highlighting that a well-defined reward function is critical for RL training in classification problems.
Reinforcement Learning and Robotics
Research in this cohort focused on solving sparse reward problems and optimizing decision-making processes.
Curiosity-Driven Robotic Manipulation
Jonathan Michaux used intrinsic motivation—specifically curiosity-driven exploration—to solve robotic tasks where extrinsic rewards are sparse or missing. By formulating the intrinsic reward as the error in the agent's ability to predict its next state given its current state and action, the project successfully solved several difficult robotic manipulation tasks in simulation.
Optimizing Medical Treatment
Elynn Chen developed a system using historical electronic health records (EHR) to recommend optimal therapeutic treatments, specifically the dosage of IV fluids and vasopressors. Using policy iteration and tabular Q-learning, the results showed that the optimal RL policies recommended lower doses of IV fluids and higher doses of vasopressors compared to actual physician treatments. Off-policy evaluation indicated that Q-learning provided higher rewards than policy iteration.
Analysis of the Discount Factor in DQN
Yuhao Wan investigated the role of the discount factor (gamma) in Deep Q-Networks (DQN). The project identified that the discount factor encodes both intertemporal preference and confidence in bootstrapping. This led to the development of a "myopia scheme" that improved baseline performance in customized Gridworld environments and proved robust in more general settings beyond the DQN framework.
Model Interpretability and Educational Applications
Two projects focused on the transparency of neural networks and the practical application of ML in education.
Visualizing GANs with Activation Atlases
Janet Brown used the techniques of Activation Atlases to evaluate and understand image synthesis Generative Adversarial Networks (GANs). This approach allowed for the measurement of differences between real and fake images in both numerical and highly visual terms, providing a window into the "black box" of what the neural network perceives.
Automated Project Labeling for Education
Nancy Otero developed CREATURE, a proof-of-concept model designed to label online projects for teachers. Because project-based learning is effective but finding appropriately labeled projects is difficult, CREATURE provides a solution by labeling online projects with 75–90% accuracy.