NeurIPS 2025 E2LM Competition: Early Training Evaluation of Language Models
Hugging Face and affiliated institutions have announced the NeurIPS 2025 E2LM (Early Training Evaluation of Language Models) competition. The initiative aims to create new benchmarks that can effectively capture reasoning and scientific knowledge signals in Large Language Models (LLMs) during the early stages of training, specifically when models have been trained on approximately 200 billion tokens or fewer.
The Challenge of Early-Stage Evaluation
Existing evaluation benchmarks often fail to provide discriminative signals during the initial phases of LLM development. When researchers conduct ablation experiments to test architectures, data mixtures, and hyperparameters, they typically rely on training loss curves and evaluation scores. However, these metrics often lack the necessary sensitivity to provide conclusive insights when models are in the early training stages (up to ~200B tokens).
Competition Framework and Participation
Participants are tasked with building benchmarks using the lm-evaluation-harness library. Submissions are managed through a dedicated Hugging Face organization and a Hugging Face Space, where an active leaderboard tracks performance.
To lower the barrier to entry, the competition utilizes Small Language Models (SLMs) that are runnable on free-tier Google Colab GPUs. A comprehensive starting kit, including notebooks, is provided to help participants begin their submissions.
Evaluation Metrics and Scoring
The global score for each submission is a weighted sum of three primary metrics, designed to prioritize signal quality and scientific accuracy:
- Signal Quality Score (Score SQ): Weighted at 0.5.
- Compliance with Scientific Knowledge Score (Score CS): Weighted at 0.4.
- Ranking Consistency Score (Score RC): Weighted at 0.1.
Global Score Formula: Score = 0.5 * Score SQ + 0.1 * Score RC + 0.4 * Score CS
Validation Procedures
To ensure the integrity of the results, all submissions undergo two validation checks:
- Scientific Alignment: Verification that the benchmark aligns with established scientific knowledge domains.
- Leakage Detection: Checking for the presence of the answer within the question prompt to prevent information leakage.
Checkpoint Accessibility
To prevent participants from over-fitting their solutions to specific models, the competition uses a tiered access system for checkpoints:
- Local Computation: Participants can compute the Signal Quality subscore locally using checkpoints for three SLMs (0.5B, 1B, and 3B parameters) trained from 0 to 200 BT.
- Hidden Checkpoints: The Ranking Consistency and other subscores are computed automatically via the Hugging Face Space using hidden checkpoints, including models trained from 200 GT to 1 T tokens and a 0.5B parameter model trained exclusively on web data.
Competition Timeline
| Phase | Dates |
|---|---|
| Kick-off | 14 July 2025 |
| Warm-up Phase | 14 July 2025 - 17 August 2025 |
| Development Phase | 18 August 2025 - 26 October 2025 |
| Final Phase | 27 October 2025 - 03 November 2025 |
| Results Announcement | 04 November 2025 |
| Winners' Fact Sheets & Code Release | 22 November 2025 |
| NeurIPS Workshop Presentation | 6 or 7 December 2025 |
Prizes and Awards
Cash prizes are awarded for the top three overall placements and specific student achievements:
- 1st Place: 6,000 USD
- 2nd Place: 4,000 USD
- 3rd Place: 2,000 USD
- Student Awards: Two awards of 2,000 USD each for the top two student-verified solutions.