OpenAI Meta-learning for Wrestling

OpenAI has developed a meta-learning agent for simulated robot wrestling that can rapidly adapt its strategy to defeat stronger opponents and compensate for physical malfunctions. This capability allows agents to optimize their performance in real-time based on a small number of interactions with a new environment or opponent.

Modified MAML for Multi-Agent Adaptation

OpenAI extended the Model-Agnostic Meta-Learning (MAML) algorithm to optimize against pairs of environments rather than single ones. While standard MAML initializes policies so they can be quickly adapted to a new task with minimal parameter updates, this modified version allows agents to adapt to previously unseen environments, provided they do not diverge too wildly from previous training data.

During execution, the agent performs gradient ascent steps on the rewards collected during the first few episodes of interaction with a new environment. This process allows the agent to refine its policy parameters on the fly to better counter a specific opponent.

Experimental Setup and Results

To test continuous adaptation, OpenAI utilized three types of simulated agents with varying morphologies:

• Ant (4-legged)
• Bug (6-legged)
• Spider (8-legged)

These agents competed in a multi-round game where they played several matches against the same opponent. The meta-learning agents adapted their policy parameters between rounds to counter the opponent's specific strategy. OpenAI found that agents capable of adapting their tactics were significantly more successful competitors than those with fixed policies.

Adaptation to Physical Malfunction

Beyond adapting to external opponents, the meta-learning approach enables agents to handle internal changes. OpenAI demonstrated that agents could adapt to physical malfunctions, such as the loss of functionality in some of their own limbs over time. This suggests that meta-learning can be used to develop agents capable of handling both external environmental shifts and changes to their own internal states or physical bodies.

Availability and Research Context

This work is part of OpenAI's broader research into large-scale multi-agent systems. To facilitate further experimentation, OpenAI has released the MuJoCo environments and the trained policies used in the research via GitHub.