Artificial Analysis Text to Image Leaderboard and Arena Launch

Hugging Face has introduced the Artificial Analysis Text to Image Leaderboard and Arena, a human-preference-based ranking system designed to compare the quality of leading open-source and proprietary image generation models. This initiative provides a scalable way to evaluate model performance using crowdsourced human judgment, filling a gap left by traditional objective metrics.

Human-Preference Methodology and ELO Scoring

The Artificial Analysis Text to Image Leaderboard utilizes a crowdsourcing approach to gather human preference data at scale. Because image quality preferences are inherently variable, the project uses an Image Arena where participants are presented with a prompt and two generated images, selecting the one that best reflects the prompt.

Key technical details of the methodology include:

  • ELO Calculation: Model rankings are determined by an ELO score derived from a regression of all human preferences, a method similar to the one used by Chatbot Arena.
  • Data Volume: The ELO scores are informed by over 45,000 human image preferences.
  • Evaluation Breadth: To ensure a wide range of use-cases, more than 700 images are generated for each model across diverse categories, including nature, animals, art, human portraits, and groups of people.

Current Market Insights and Model Performance

Early results from the leaderboard highlight a narrowing gap between proprietary and open-source image generation models.

Proprietary vs. Open Source

Proprietary models, specifically Midjourney, Stable Diffusion 3, and DALL·E 3 HD, currently lead the leaderboard. However, open-source models are becoming increasingly competitive; Playground AI v2.5 currently leads the open-source category and surpasses OpenAI’s DALL·E 3 in some rankings.

Rapid Evolution of the Field

The speed of advancement in image generation is evidenced by the decline of previous leaders. DALL·E 2, which was a clear leader a year ago, is now selected in the arena less than 25% of the time and ranks among the lowest models.

Impact of Stable Diffusion 3 Medium

Stable Diffusion 3 is currently a top contender on the leaderboard. The announcement that Stable Diffusion 3 Medium would be open-sourced on June 12 is expected to significantly benefit the open-source community, potentially leading to a surge in community-driven fine-tuned versions, similar to the trajectory of Stable Diffusion 1.5 and SDXL.

Community Participation and Tooling

The project is hosted as a Space on Hugging Face, offering two primary ways for users to interact with the data:

  • The Leaderboard: A public ranking of the leading image models.
  • The Image Arena: A participatory interface where users contribute preferences. After 30 votes, users can access a "Personal Leaderboard" to see their own personalized model rankings based on their specific preferences.

Sources