linzhiqiu/t2v_metrics

Evaluating text-to-image/video/3D models with VQAScore

What it solves

VQAScore is designed to automatically evaluate the quality and alignment of text-to-visual generative models (images, videos, and 3D). It replaces traditional metrics like CLIPScore with a more robust approach that uses image-to-text generation to score how well a visual output matches a given text prompt.

How it works

The framework leverages large multimodal models (LMMs) to calculate the probability of a specific answer (e.g., "Yes") given a visual input and a question about the prompt. It supports a wide range of state-of-the-art models, including GPT-4o, Gemini 2.5, Gemma 3, and the Qwen family (VL, 3.5, and Omni).

For open-source Qwen models, it provides forward_with_trace for token-level transparency, allowing users to see the exact probabilities and alternatives the model considered when assigning a score.

Who it’s for

AI researchers and developers building generative visual models who need a standardized, automated way to measure how accurately their models follow complex prompts, particularly for compositional text-to-visual generation.

Highlights

  • Multimodal Support: Evaluates images, videos, and audio-visual alignment (via Qwen3-Omni).
  • Broad Model Integration: Compatible with both proprietary APIs (GPT-4o, Gemini via Vertex AI) and open-weights models (Qwen, Gemma, PaliGemma).
  • Flexible Templating: Allows users to customize question and answer templates to change how the scoring is performed.
  • Benchmarking Tools: Includes integration with GenAI-Bench for evaluating compositional generation.