Hugging Face ICML 2026 Open Reproductions Report

Hugging Face organized a community-driven hackathon from July 15 to August 2, 2026, utilizing coding agents to reproduce claims from the ICML 2026 conference. The effort resulted in the reproduction of 2,226 papers—approximately one-third of the conference—generating 6,816 Trackio logbooks and judging 35,908 individual claims.

Large-Scale Reproducibility Findings

An analysis of claim-level verdicts across the examined papers revealed a significant split in reproducibility:

Verified Claims: 51% of examined papers (1,103) had at least one independently verified claim. Within this group, 266 papers were fully reproduced (all claims verified) and 632 were partially reproduced with no falsifications. A total of 3,978 individual claims were confirmed via experiments.
Falsified or Contested Claims: 23% of examined papers (496) had at least one claim falsified or contested. This includes 49 papers where every claim was falsified and 242 papers where different reproduction teams reached opposite verdicts on the same claims.
Inconclusive or Toy Results: 502 papers provided only toy-scale evidence, and 280 papers remained inconclusive, often due to missing artifacts or proprietary datasets.

Technical Analysis of Confirmed Falsifications

Hugging Face adversarially re-verified 35 formal falsification claims. Several high-profile errors were identified and confirmed:

Mathematical and Proof Errors

In the paper "Towards Optimal Robustness in Learning-Augmented Paging," a claim of robustness $H_{k} + O(1)$ was falsified. Reproduction efforts found the additive term grew as $0.38 \ln k$, meaning the true robustness is $H_{k} + \Theta(\log k)$.

In "Attention's forward pass and Frank-Wolfe," a theorem regarding token particle collapse was found to fail at step 224 (and later at 3,800 and 6,416 steps). These violations were missed by other reviewers because finite-horizon checks stopped too early.

Implementation Discrepancies

In "Self-Distillation Enables Continual Learning," the theoretical analysis focused on reverse KL divergence, but the released code—which produced the paper's results—used forward KL. Additionally, the paper's headline +4pp result could not be reproduced using the authors' own code and data.

Evaluation Flaws

In "Do Transformers Need Three Projections?", it was discovered that approximately 66% of evaluated label positions were EOS padding tokens. This padding deflated perplexity threefold, changing the reported "3.1% quality cost for 50% cache reduction" to approximately 9.4%.

The Role of Coding Agents in Research Audit

The hackathon utilized a variety of agent frameworks, including Claude Code, Codex, Cursor, and OpenResearch's orx. Each reproduction produced a Trackio logbook containing the write-up, code, artifacts, and agent execution traces.

Agent Limitations

Despite their efficiency, agents encountered several failure modes:

Local Loops: Agents occasionally became stuck in repetitive execution cycles.
Scale Misinterpretation: Some agents verified claims that were actually false because they stopped experiments before scale-dependent behavior (such as the $\log k$ growth in the paging paper) became apparent.
Unit Mismatches: Some falsifications were based on incorrect unit assumptions by the agent.

Human-in-the-Loop Necessity

The most reliable results emerged from workflows where humans steered the agents. For example, in a reproduction of a paper on stable image generation under extreme quantization, numerical metrics showed no collapse, but a human was required to perceptually judge 128 image pairs to determine if the images were actually usable.

Hackathon Methodology

To facilitate the audit, Hugging Face indexed 6,341 accepted ICML 2026 papers and extracted core scientific claims to provide agents with concrete targets. The evaluation process involved:

Agent Execution: Participants used their chosen agents to write code and run experiments, utilizing HF Jobs for compute.
Logbook Generation: Every run produced a static Hugging Face Space (Trackio logbook) for transparency.
Automated Judging: An open-weights model (GLM-5.2) served as the Logbook Judge, issuing verdicts of verified, falsified, toy, or inconclusive for each claim, treating self-assessments as untrusted.