Learning to cooperate, compete, and communicate – OpenAI 2017 research announcement

TL;DR

OpenAI released the MADDPG algorithm for centralized learning and decentralized execution in multiagent reinforcement learning, showing that agents can learn to cooperate, compete, and communicate better than with prior methods.

Introduction

Multiagent environments where agents compete for resources are considered stepping stones toward AGI. They provide a natural curriculum—difficulty matches the skill of competitors—and have no stable equilibrium, creating constant pressure to improve.

Where traditional RL struggles

Traditional decentralized RL approaches such as DDPG, actor‑critic, and deep Q‑learning struggle in multiagent settings because each agent must try to predict others’ actions while also acting itself, especially in competitive scenarios. Policy gradient methods suffer from high variance, and adding a critic improves stability but still fails to solve tasks like cooperative communication. Considering the actions of others during training is important for learning collaborative strategies.

MADDPG algorithm

We developed MADDPG (Multi‑Agent Deep Deterministic Policy Gradient) to enable centralized learning and decentralized execution. The algorithm extends DDPG by borrowing from actor‑critic techniques.

  • Each agent is treated as an "actor" that receives advice from a "critic" to decide which actions to reinforce during training.
  • In standard RL the critic predicts the value (expected future reward) of an action in a given state, which the actor uses to update its policy.
  • To make multiagent learning feasible, we enhance the critic so it can access the observations and actions of all agents, as shown in the diagram.
  • At test time agents do not need the central critic; they act based on their own observations and their predictions of other agents’ behavior.
  • Because a centralized critic is learned independently for each agent, the approach can model arbitrary reward structures, including adversarial cases where rewards oppose each other.

Experimental results

We tested MADDPG on a variety of tasks and found it performed better than DDPG on all of them. The accompanying animations illustrate:

  • Two AI agents learning to go to a specific location while splitting up to hide their intended location from an opposing agent.
  • One agent communicating the name of a landmark to another agent.
  • Three agents coordinating to travel to landmarks without colliding.

Initial research challenges

Before MADDPG, using decentralized techniques we observed that listener agents often learned to ignore the speaker when it sent inconsistent messages about where to go. The listener would set all weights associated with the speaker’s message to zero, effectively deafening itself. Once this happened, recovery was hard because the speaker received no feedback about whether its messages were correct.

We tried a technique from a recent hierarchical reinforcement project that forced the listener to incorporate the speaker’s utterances, but this only made the listener pay attention without helping the speaker learn what to say that is relevant.

The centralized critic in MADDPG addresses this by helping the speaker learn which utterances might be relevant to the actions of other agents.

Next steps

Agent modeling has a long history in AI research, with prior work focusing on games with few time steps and small state spaces. Deep learning now handles complex visual inputs, and RL provides tools for learning behaviors over long periods.

With MADDPG we can train multiple agents simultaneously without requiring them to know the environment’s dynamics, opening the door to a broader set of problems involving communication, language, and learning from high‑dimensional information.

Researchers interested in evolving agents are invited to consider opportunities at OpenAI.

Authors

Ryan Lowe, Igor Mordatch, Pieter Abbeel, Yi Wu, Aviv Tamar, Jean Harb

Sources