SeekingDream/Static-to-Dynamic-LLMEval

What it solves

Large language models (LLMs) are trained on huge amounts of text scraped from the Internet. Sometimes, the questions and answers used for testing a model accidentally end up in that training data. This is called data contamination, and it makes benchmark scores look better than they really are. Traditional static benchmarks – fixed sets of questions – are especially vulnerable because the exact same test items can appear in training data. This survey addresses that problem by reviewing how the field is moving from static to dynamic benchmarks, which generate fresh test items or change over time to reduce contamination.

How it works

The repository is a survey paper, not a software tool. It systematically reviews published research on LLM evaluation. It first looks at ways people have tried to make static benchmarks safer – such as canary strings, encryption, label protection, and post-hoc detection – and explains why each method has limits. Then it examines dynamic benchmarks, which avoid contamination by creating new test material on the fly. The survey categorizes dynamic approaches into temporal cutoff, rule-based generation (template-based, table-based, graph-based), LLM-based generation (benchmark rewriting, interactive evaluation, multi-agent evaluation), hybrid generation, and network & system methods. Based on this review, the authors propose a set of design principles for dynamic benchmarks and point out that there are currently no standard criteria for evaluating such benchmarks.

Who it's for

This is primarily for researchers and engineers who build or use LLM benchmarks. If you are developing evaluation suites, designing tests to measure model capabilities, or worried that your model's reported scores are inflated due to leaked test data, this survey gives a structured overview of the current landscape and open problems.

Highlights

• Provides a systematic taxonomy of static and dynamic benchmarking methods for data contamination.
• Covers both mitigation techniques for static benchmarks (canary strings, encryption, label protection, post-hoc detection) and a wide range of dynamic benchmarking approaches.
• Highlights a critical gap: the lack of standardized evaluation criteria for dynamic benchmarks.
• Proposes optimal design principles for dynamic benchmarking based on analysis of existing methods.
• Includes a large curated list of relevant papers with links, and welcomes community contributions via pull requests.