LAVE: Zero-shot VQA Evaluation on Docmatix with LLMs
Hugging Face has introduced LAVE (LLM-Assisted VQA Evaluation), a metric that uses Large Language Models (LLMs) to evaluate Visual Question Answering (VQA) performance. This approach addresses the failure of traditional exact-match metrics to recognize semantically correct answers in out-of-distribution (OOD) and zero-shot settings, specifically when using synthetic datasets like Docmatix.
The Failure of Traditional VQA Metrics in OOD Settings
Traditional VQA evaluation relies on exact string matching between predicted and reference answers, which is effective for independent and identically distributed (IID) data but fails in out-of-distribution (OOD) scenarios. In zero-shot transfer tasks, models often generate answers that are semantically correct but differ in format, specificity, or interpretation from the human-curated reference answers.
This discrepancy was observed during the development of Docmatix, where fine-tuning Florence-2 on the dataset yielded high semantic performance but low benchmark scores. Further fine-tuning on the DocVQA benchmark improved the scores by teaching the model the required syntax, but human evaluators found the resulting model performed worse, indicating that traditional metrics do not always align with human perception.
The LAVE Evaluation Method
LAVE frames VQA evaluation as an answer-rating task using in-context learning with an LLM. Instead of binary matching, it uses a nuanced rating scale to account for ambiguity and incompleteness.
Rating Scale and Prompting
LAVE utilizes a rating scale from 1 to 3:
- 1: Incorrect or irrelevant answer.
- 2: Ambiguous or incomplete answer.
- 3: Correct answer.
To ensure high-quality ratings, the LLM is prompted to provide a rationale before the final rating and is strictly instructed to provide only one rating. For binary questions, the prompt explicitly mandates that only 'yes' or 'no' are acceptable for a correct rating.
Scoring Function
The final rating ($r$) is extracted from the last character of the LLM's response and mapped to a score ($s$) in the range [0, 1] using the formula:
$s = \frac{r - 1}{2}$
Experimental Results on Docmatix
To test LAVE, Hugging Face used MPLUGDocOwl1.5 as a baseline model (which achieves an 84% ANLS score on the original DocVQA test subset) and performed zero-shot generation on a 200-image subset of Docmatix. Llama-2-Chat-7b was employed as the rating LLM.
Quantitative Comparison
Comparing LAVE against traditional metrics reveals a significant gap in how model performance is perceived:
| Metric | Score |
|---|---|
| CIDER | 0.1411 |
| BLEU | 0.0032 |
| ANLS | 0.002 |
| LAVE | 0.58 |
Key Takeaways
The use of LLMs for evaluation resulted in an approximately 50% accuracy gain over traditional metrics. This suggests that current VQA evaluation standards are overly rigid and may penalize models that provide correct information in a non-standard format. The findings indicate a need for evaluation metrics that better align with human judgment, especially when assessing zero-shot performance on synthetic datasets.