Karine-Huang/T2I-CompBench

What it solves

T2I-CompBench (and its enhanced version T2I-CompBench++) is a comprehensive benchmark designed to evaluate how well text-to-image (T2I) generation models can handle compositional elements. It addresses the challenge of ensuring that AI-generated images accurately reflect complex prompts, specifically focusing on attribute binding, spatial relationships, and numeracy.

How it works

The project provides a dataset and a suite of evaluation metrics to test T2I models. It uses several specialized tools to measure accuracy:

• BLIP-VQA: Used for evaluating attribute binding (e.g., ensuring a "green bench" is actually green).
• UniDet: Used for 2D/3D spatial relationships and numeracy (counting objects).
• CLIPScore: Used for non-spatial relationships.
• MLLM Evaluation: Supports using Multimodal Large Language Models (MLLMs) like GPT-4V, MiniMax M2.7, and ShareGPT4V-CoT to score image-text alignment.
• GORS Finetuning: Includes a method for finetuning diffusion models (via LoRA) to improve their compositional performance based on reward-based feedback.

Who it’s for

• AI Researchers: Those developing new text-to-image models or improving existing ones.
• Model Evaluators: People needing a standardized way to measure if a model can correctly place objects and bind attributes to the correct items.

Highlights

• Industry Adoption: Its evaluation metrics have been adopted by major models including DALL-E 3, Stable Diffusion 3, and PixArt-α.
• Comprehensive Scope: Covers attribute binding, 2D/3D spatial relations, numeracy, and complex compositions.
• MLLM Integration: Integrates modern vision-language models for automated, high-quality evaluation.