Learning to communicate: OpenAI's 2017 research on emergent AI language

Overview

OpenAI's 2017 'Learning to communicate' research demonstrates that reinforcement‑learning agents can invent a grounded, compositional language when rewarded for cooperative goals.

Training Setup

Agents are placed in a two‑dimensional world with simple landmarks and given individual goals that are best achieved by communicating with other agents. Each agent can broadcast a message to the group at each time step, and receives a reward equal to the sum of all agents’ rewards, encouraging collaboration. The agents choose between environment actions (moving or looking) and communication actions (broadcasting a word‑like symbol). They observe previous communications and the locations of all entities, storing the observed messages in a private recurrent neural network.

Emergent Language Properties

The resulting language is both grounded and compositional. Grounded means each word is tied to something the agent directly experiences in its environment, such as associating a symbol with a landmark or another agent. Compositional means agents can combine multiple words into a message to represent a specific idea, for example instructing another agent to move to a particular location.

Overcoming Challenges

Early experiments produced idiosyncratic communication patterns that hindered interpretability. First, agents tended to repeat a single utterance with spaces, creating a Morse‑code‑like, non‑compositional language; adding a small cost per utterance and a preference for quick task completion encouraged concise use and a larger vocabulary. Second, when given large vocabularies, agents sometimes encoded entire sentences into a single word, which would require exponential vocabulary growth; this was discouraged by rewarding the use of frequently spoken words, promoting a compact vocabulary. Third, agents initially referred to landmarks using spatial terms like “top‑most” or “left‑most” that depended on a global coordinate frame; switching to an ego‑centric frame eliminated these directions and led agents to refer to landmarks by their color property.

Non‑verbal Communication

The same training framework works when agents cannot exchange textual messages. In those settings agents improvise physical actions such as pointing, guiding, or pushing a sightless teammate toward a goal, as shown in the accompanying animations.

Grounded vs Symbolic Language

Unlike approaches that train language models on massive text corpora, which lack a direct link to the world and resemble the Chinese Room thought experiment, this method grounds language in the agents’ perceptual experience, aiming for a deeper understanding of how words relate to reality.

Future Directions

The team plans to increase environmental complexity and the range of actions available to agents to foster more expressive languages beyond basic verbs and nouns. As the invented language grows richer, interpreting it will become harder for humans, so a forthcoming project will connect the emergent language with English by having agents communicate with English‑speaking partners, thereby automating translation. This interdisciplinary effort will involve collaboration with researchers at UC Berkeley and invites those interested in smarter language models to consider careers at OpenAI.

For More Information

The initial results are described in the accompanying arXiv preprint.