Hugging Face AI vs. AI: Multi-Agent Reinforcement Learning Competition System

Hugging Face has launched AI vs. AI, an open-source system designed to rank the strength of deep reinforcement learning (RL) models in multi-agent settings. By facilitating continuous competition between agents, the system provides a relative measure of skill and a robust evaluation technique that tests policies against a diverse range of behaviors.

System Architecture and Components

AI vs. AI is composed of three primary technical elements hosted on Hugging Face infrastructure:

  • Hugging Face Space: Utilizes a matchmaking algorithm and a Scheduler to run model fights as background tasks on free hardware.
  • Hugging Face Dataset: Serves as the persistence layer, storing match history and model ratings.
  • Leaderboard: A public interface that retrieves match results from the dataset to display the current ELO ratings of the models.

When a user pushes a trained model to the Hub, the system automatically evaluates and ranks it against other submitted agents.

The ELO Rating and Matchmaking Algorithm

Rather than relying on objective metrics, AI vs. AI uses the ELO rating system to establish a relative measure of skill. In this implementation, the system starts all new models at an arbitrary rating of 1200. To maintain a constant average ELO across the pool, gains and losses are symmetrical (e.g., +10 and -10).

Matchmaking Process

The matchmaking algorithm ensures a balance between diversity and competitive parity through the following steps:

  1. Model Gathering: All available models are collected from the Hub.
  2. Queue Creation: Models are placed into a queue.
  3. Pairing: The system pops the first model and pairs it with a random model chosen from the $n$ models with the closest ratings.
  4. Simulation: The match is simulated within the environment (such as a Unity executable), and results are recorded in a Hugging Face Dataset.
  5. Rating Update: The ELO formula is applied to update the ratings of both participants based on the outcome.
  6. Iteration: The process repeats until the queue is empty, after which the cycle restarts.

Implementation: The SoccerTwos Challenge

To demonstrate the system, Hugging Face implemented the SoccerTwos Challenge as part of Unit 7 of their Deep Reinforcement Learning Course. This challenge utilizes the Unity ML-Agents environment, where 2vs2 soccer teams must collaborate to score goals.

Key features of this implementation include:

  • Visual Demo: A dedicated Space allowing users to select two teams and visualize their match in real-time.
  • Community Engagement: A dedicated Discord channel (ai-vs-ai-competition) for sharing advice and exchanging insights.

Generalization and Future Applications

Because the AI vs. AI system is environment agnostic, it can be applied to any adversarial multi-agent setting. Hugging Face intends to expand the tool to support other environments, including those from PettingZoo and custom environments like "SnowballFight."

By testing an agent against a wide variety of policies, the system serves as a robust evaluation technique, providing a comprehensive view of policy quality that traditional static metrics may fail to capture.

Sources