pathwaycom/arc-task-gen

What it solves

This project addresses the problem of data contamination in AI benchmarks. Because the public ARC-AGI-1 dataset is widely available, it is difficult to determine if a model is truly reasoning or simply recalling tasks it has seen during training. This tool generates new, distribution-matched tasks that resemble the public dataset but are unseen by models, allowing for a more honest evaluation of a few-shot rule induction capability.

How it works

The tool creates a private evaluation set of tasks in the standard ARC format (a tasks.json file containing training and test pairs). These generated tasks maintain the same properties as the original ARC-AGI-1 dataset, making them compatible with existing evaluation harnesses while ensuring the models are tested on novel problems.

Who it’s for

Researchers and developers building reasoning models—such as the BDH-CQ architecture—who need to verify that their model's performance on the ARC-AGI-1 benchmark is due to genuine reasoning rather than memorization.

Highlights

• Distribution-Matched: Generates tasks that mirror the characteristics of the public ARC-AGI-1 set.
• Benchmark Compatibility: Produces output in the standard ARC JSON format for easy integration with existing tools.
• Contamination-Free: Provides a way to create private evaluation sets to isolate few-shot rule induction.
• Independent Validation: Used to evaluate the BDH-CQ model, which was independently reproduced by experts including a co-author of the Transformer architecture.