Improving Prompt Consistency with Structured Generations
Structured generation—constraining a model's output to a specific format via regular expressions or context-free grammars—significantly reduces the sensitivity of LLM benchmarks to prompt formatting and shot ordering. This approach not only stabilizes model rankings but often increases overall expected performance, addressing a critical flaw in current LLM evaluation where minuscule changes in prompt structure can lead to disparate results.
The Problem: Evaluation Sensitivity to Prompt Format
LLM benchmark performance is highly dependent on the specific format of the prompt, even when the underlying information remains identical. Research by the Hugging Face Leaderboards and Evals team revealed that superficial changes to prompt formatting can lead to wide variations in accuracy and unstable model rankings.
Impact on Accuracy and Ranking
In experiments using the MMLU task across five different models and eight prompt formats, the team observed the following:
- Score Variance: Model performance varied by approximately 10 points across different formats. In one extreme case, Qwen1.5-7B's accuracy dropped from 51.2% to 22.9% due to a tokenizer issue associated with a specific prompt variation.
- Ranking Instability: No model remained consistently ranked across all prompt variations. This instability suggests that model authors could potentially manipulate reported results by selecting the prompt format most advantageous to their specific model.
- Shot Order Sensitivity: Even when the prompt format and examples are identical, simply shuffling the order of few-shot examples can result in performance differences of up to 3 points.
Structured Generation as a Solution
To combat this variance, Hugging Face and Dottxt explored using structured generation via the Outlines library. Instead of focusing on the input (the prompt), this method focuses on the output by forcing the model to follow a predefined structure.
Technical Implementation
For tasks like GSM8K (grade school math word problems), the researchers implemented a regular expression to constrain the output. The regex required the model to:
- Reason for between 200 and 700 characters.
- Explicitly state "The answer is".
- Provide a number up to 10 digits long (not starting with 0).
This technique, referred to as "thought control," allows researchers to regulate the amount of reasoning the model performs before arriving at an answer.
Experimental Results
GSM8K: N-Shot Prompting
Testing Mistral-7Bv0.1 and Zephyr-7B-beta on the GSM8K dataset with 1- to 8-shot prompting showed that structured generation:
- Reduced Variance: Performance variance across different n-shot setups was significantly lowered.
- Stabilized Ranking: The relative ranking of the models remained consistent; Mistral consistently outperformed Zephyr.
- Improved Efficiency: 1-shot structured performance was substantially better than 1-shot unstructured performance, performing on par with 5-shot unstructured generation.
GPQA: N-Shot and Shot Order Variations
Using the Graduate-Level Google-Proof Q&A Benchmark (GPQA) 'diamond' subset, the researchers tested the impact of varying both the number of shots and the order of those shots (using different seeds for shuffling).
Performance Gains:
| Model | Unstructured Mean | Structured Mean |
|---|---|---|
| Mistral-7B-v0.1 | 0.2360 | 0.2935 |
| Zephyr-7b-beta | 0.2387 | 0.3048 |
Variance Reduction:
| Model | Unstructured Std Dev | Structured Std Dev |
|---|---|---|
| Mistral-7B-v0.1 | 0.0213 | 0.0202 |
| Zephyr-7b-beta | 0.0273 | 0.0180 |
Conclusion on Ranking Consistency
Across the GPQA experiments, the application of structured generation led to a major improvement in the consistency of determining a "winner" between the two models. By constraining the output, the researchers minimized the impact of spurious prompt features, making the evaluation a more accurate reflection of the model's actual capabilities rather than its sensitivity to prompt formatting.