Can Foundation Models Label Data Like Humans? Hugging Face Analysis
Hugging Face researchers investigated whether foundation models, specifically GPT-4, can reliably label data and evaluate model preferences as a proxy for human judgment. The study reveals that while GPT-4 can mirror some human rankings, it suffers from significant positional bias and a preference for verbosity over conciseness, making it an imperfect substitute for human annotators.
LLM Preference Evaluation Methodology
Hugging Face expanded the Open LLM Leaderboard to include a controlled study comparing human labels (via Scale AI) and GPT-4 evaluations. The researchers used a held-out set of 327 high-quality, human-written prompts across categories such as generation, brainstorming, question answering, summarization, commonsense, and coding.
Four open-source models were evaluated: Vicuna-13B, Koala-13B, OpenAssistant-12B, and Dolly-12B. Raters (both human and GPT-4) used a Likert scale from 1 to 8 to indicate preference between two model completions, where 1 represents a strong preference for the first model and 8 for the second.
Human vs. GPT-4 Elo Rankings
Both human and GPT-4 evaluators produced similar relative rankings of the models, but the margins differed. In both sets of results, Vicuna-13B consistently ranked highest, followed by Koala-13B, OpenAssistant-12B, and Dolly-12B.
Human Elo Results (Median)
| Model | Elo Ranking (No Ties) | Elo Ranking (With Ties) |
|---|---|---|
| Vicuna-13B | 1140 | 1130 |
| Koala-13B | 1073 | 1061 |
| Oasst-12B | 986 | 988 |
| Dolly-12B | 802 | 820 |
GPT-4 Elo Results (Median)
| Model | Elo Ranking (No Ties) | Elo Ranking (With Ties) |
|---|---|---|
| Vicuna-13B | 1134 | 1114 |
| Koala-13B | 1082 | 1082 |
| Oasst-12B | 972 | 973 |
| Dolly-12B | 812 | 831 |
Critical Biases in LLM-Based Evaluation
The study identified several systemic biases that affect the reliability of GPT-4 as an evaluator:
Positional Bias
GPT-4 exhibits a strong positional bias, frequently favoring the first response presented in a pairwise comparison (rating it as "1" on the 1-8 scale), regardless of the actual quality of the content.\n
Verbosity and Style Bias
GPT-4 tends to prefer longer, more detailed responses over concise, correct ones. This is evidenced by examples where GPT-4 rated a verbose model response as "greatly better" than a brief, accurate human response. This suggests a "style benefit" for models trained on outputs from other large language models (e.g., Vicuna), which the researchers describe as "unintentional doping."
Human Demonstration Paradox
When human-written demonstrations were included in the evaluation, GPT-4 ranked them lower than several open-source models (e.g., Vicuna and Koala), further highlighting the model's preference for LLM-style verbosity over human-style conciseness.
Correlation and Task Performance
The correlation between GPT-4 and human labels varies significantly by task type. GPT-4 is more aligned with humans in high-entropy, creative tasks than in low-entropy, factual or technical tasks.
| Category | Correlation: GPT-4 to Human Labels |
|---|---|
| Brainstorm | 0.60 |
| Creative generation | 0.55 |
| Commonsense reasoning | 0.46 |
| Question answering | 0.44 |
| Summarization | 0.40 |
| Natural language to code | 0.33 |
Key Takeaways for LLM Evaluation
- Positional Bias is Persistent: Asking GPT-4 to "de-bias" itself often simply flips the bias in the opposite direction rather than eliminating it.
- Verbosity $\neq$ Quality: GPT-4 rewards token quantity, which can lead to misleading rankings where verbose but less useful models outperform concise and accurate ones.
- Task-Dependent Reliability: LLM-based evaluation is most reliable for brainstorming and creative generation and least reliable for coding tasks.
- Formatting Limitations: The researchers noted that models like ChatGPT often struggle to return answers in the correct format for Likert scales, indicating that formatting control is a prerequisite for becoming a useful evaluation tool.