BenchMIRT: Auditing LLM Benchmarks with Multidimensional Item Response Theory

BenchMIRT is a new method for auditing Large Language Model (LLM) benchmarks at the level of individual prompts to determine which underlying capabilities are actually driving a benchmark's score. By analyzing model performance on a per-question basis, BenchMIRT allows researchers to separate mixed signals—such as safety and general reasoning—that are often collapsed into a single aggregate score.

Multidimensional Item Response Theory (MIRT)

BenchMIRT is based on Multidimensional Item Response Theory (MIRT), a psychometric technique used to measure traits and abilities from patterns of test responses. Unlike single-dimensional IRT, which assumes a test measures one trait, MIRT can separate multiple capabilities that contribute to a model's performance on the same set of questions.

BenchMIRT operates at two levels:

  • Model Level: It estimates a model's strength in the specific capabilities reflected in the selected benchmarks.
  • Question Level: It estimates the difficulty of a question and its ability to distinguish between models that are stronger or weaker in those specific capabilities.

To validate the method, the researchers trained BenchMIRT on results from 100 LLMs across 16 benchmarks (including MMLU-Pro, GPQA, MATH, BBH, HarmBench, StrongReject, WildJailbreak, BBQ, WMDP, and XSTest) and over 34,000 questions. Without being told which benchmarks measured which capabilities, BenchMIRT independently and stably recovered two dominant dimensions: safety and general reasoning.

Disentangling Benchmark Signals

BenchMIRT reveals that many benchmarks measure a combination of capabilities rather than a single intended metric. While some benchmarks aligned with their intended focus, others showed complex overlaps:

  • BBQ (Social Bias): Despite being grouped with safety benchmarks, BBQ aligned more strongly with general reasoning. This suggests that low scores on BBQ may reflect a model's difficulty in reasoning through questions rather than its safety behavior alone.
  • WMDP (Dual-Use Knowledge): BenchMIRT found that WMDP scores associated more strongly with general reasoning than safety. Because the benchmark rewards refusing dangerous knowledge, stronger general reasoning was associated with lower WMDP scores.
  • HarmBench (Harmful Requests): This benchmark mixes signals. Standard and contextual questions aligned with safety, but copyright-related questions (e.g., requesting song lyrics) aligned more closely with general reasoning.

Improving Evaluation Efficiency and Precision

By identifying which questions are most informative, BenchMIRT can be used to create more efficient evaluations without sacrificing accuracy:

  • Question Reduction: Keeping only 10% of the questions in the analyzed benchmarks generally preserved the relative ranking of models' safety or reasoning capabilities. Keeping 50% of the questions often matched the full benchmark's measure even more closely.
  • Performance Prediction: BenchMIRT can predict whether a model will answer a held-out question correctly with 79% accuracy, compared to 70% accuracy for a simple average-score baseline.

Limitations and Risks

The analysis is subject to several constraints:

  • Temporal Limit: The training and evaluation models were all released by March 2025, meaning the results may not capture the behavior of newer LLM generations.
  • Dimension Dependency: The discovered dimensions (safety and reasoning) depend on the specific set of benchmarks provided; different benchmark sets would likely surface different capabilities.
  • Security Risk: The ability to identify the most informative questions in a safety benchmark could potentially be used to remove those questions, creating a weaker evaluation that unsafe models could pass.

Despite these trade-offs, BenchMIRT provides a framework for more targeted benchmark design and more transparent interpretation of LLM capabilities.

Sources