OpenAI Teacher–Student Curriculum Learning

OpenAI has introduced Teacher–Student Curriculum Learning (TSCL), a framework designed to automate the creation of training curricula for complex tasks. By utilizing a "Teacher" algorithm to dynamically select subtasks for a "Student" model, TSCL enables agents to solve problems that are otherwise impossible to learn via direct training or uniform sampling.

The TSCL Framework

Teacher–Student Curriculum Learning operates by splitting a complex task into a set of available subtasks. The framework consists of two primary components:

  • The Student: The model attempting to learn the complex target task.
  • The Teacher: An algorithm that automatically chooses which subtasks from the given set the Student should train on at any given time.

Teacher Algorithm Logic

The Teacher algorithms in TSCL are guided by two primary intuitions to optimize the learning process:

  1. Maximizing Progress: The Teacher prioritizes subtasks where the Student is making the fastest progress, identified as the point where the slope of the learning curve is at its highest.
  2. Preventing Forgetting: To address the problem of catastrophic forgetting, the Teacher also selects subtasks where the Student's performance is actively declining.

Experimental Results and Performance

OpenAI demonstrated the effectiveness of TSCL across two distinct tasks: the addition of decimal numbers using Long Short-Term Memory (LSTM) networks and navigation within Minecraft.

Minecraft Navigation

In Minecraft maze navigation, TSCL showed significant advantages over baseline training methods:

  • Solving Unsolvable Tasks: The automatically generated curriculum allowed the model to solve a Minecraft maze that could not be solved at all when training directly on the final goal.
  • Training Efficiency: Learning was an order of magnitude faster than when using uniform sampling of subtasks.

Decimal Addition

In the task of adding decimal numbers, TSCL matched or surpassed the results achieved by carefully hand-crafted curricula, proving that an automated system can replace the need for manual curriculum design.

Implications for Automatic Curriculum Learning

TSCL demonstrates that automatic curriculum learning can outperform manual design by dynamically adjusting the difficulty and focus of training data based on real-time performance metrics. By balancing thes pursuit of rapid progress with the maintenance of previously learned skills, the framework provides a scalable approach to training models on complex, multi-stage tasks.

Sources