OpenAI Iterated Amplification for Complex Goal Learning
OpenAI has introduced iterated amplification, a proposed AI safety technique designed to specify complicated behaviors and goals that exceed human scale. This method allows for the training of AI systems on tasks that humans cannot directly perform or judge by demonstrating how to decompose a complex task into simpler, manageable sub-tasks.
The Challenge of Training Signals for Complex Tasks
Machine learning requires a training signal—such as labels in supervised learning or rewards in reinforcement learning—to evaluate performance and facilitate learning. While some tasks have algorithmic training signals (e.g., scoring a game of Go) or can be judged by humans, many real-world tasks are too complex for human performance or judgment. Examples include managing the security of a large computer network or designing a complex transit system.
Without an accurate training signal, AI systems cannot learn a task, or worse, they may develop unintended and dangerous behaviors due to faulty reward functions.
How Iterated Amplification Works
Iterated amplification generates training signals for complex tasks based on two primary assumptions: first, that a human can identify the smaller components that make up a piece of a larger task; and second, that a human can perform very small instances of that task.
The process follows an iterative cycle:
- Initial Training: The system samples small sub-tasks and is trained using demonstrations from humans who are capable of solving these small-scale problems.
- Task Expansion: The system samples slightly larger tasks. Humans help solve these by breaking them down into the smaller pieces that the AI (trained in the previous step) can already solve.
- Direct Learning: The solutions to these harder tasks, obtained through human-assisted decomposition, are used as a training signal to train the AI to solve these second-level tasks directly without human intervention.
- Iteration: This process is repeated, iteratively building up a training signal for increasingly composite tasks.
If successful, this results in a fully automated system capable of solving highly composite tasks despite the absence of an initial direct training signal.
Experimental Results in Toy Domains
Because working with beyond-human-scale tasks and actual human signals is complex for a prototype, OpenAI tested iterated amplification on five toy algorithmic domains using supervised learning. In these experiments, the researchers pretended not to know the direct algorithmic solutions, simulating a scenario where a human can combine sub-pieces of a solution but cannot provide a direct training signal.
The five tasks tested were:
- Permutation powering
- Sequential assignments
- Wildcard search
- Shortest path
- Union find
In all five cases, iterated amplification performed competitively with direct supervised learning, matching the performance of systems that had access to ground truth labels despite the amplification process having no direct access to those labels.
Relationship to Other AI Safety Methods
Iterated amplification shares conceptual goals with other OpenAI safety research but differs in implementation:
- AI Safety via Debate: Both seek to train tasks beyond human ability through iterative processes and indirect supervision, but they use different specific approaches.
- Human Feedback: Iterated amplification implements a reward prediction system and is expected to incorporate direct human feedback in future versions.
- Expert Iteration: While similar to the method used in AlphaGo Zero, expert iteration reinforces an existing training signal, whereas iterated amplification builds a training signal from scratch.