arcprize/arc-agi-benchmarking

What it solves

This project provides a standardized framework for benchmarking AI models on the ARC-AGI (Abstraction and Reasoning Corpus) tasks. It simplifies the process of running these complex reasoning tasks across various model providers, managing the technical overhead of API rate limits, retries, and performance scoring.

How it works

The system uses a set of model adapters to interface with different AI providers (such as OpenAI, Anthropic, Gemini, and others). Users define model configurations in a YAML file, specifying the model ID, provider, and parameters like temperature. The framework then executes tasks from ARC-AGI datasets, manages the outbound request pacing to avoid rate limits, and saves the model's responses as submissions. Finally, a dedicated scoring script compares these submissions against the ground-truth solutions to calculate accuracy metrics.

Who it’s for

• AI Researchers testing the general intelligence and reasoning capabilities of LLMs.
• Model Developers wanting to evaluate their models against the ARC-AGI benchmark.
• Benchmark Engineers who need a scalable way to run multiple model configurations across various datasets concurrently.

Highlights

• Multi-Provider Support: Built-in adapters for a wide range of providers including OpenAI, Anthropic, Gemini, Fireworks, Grok, and OpenRouter.
• Concurrent Execution: Ability to run multiple model configurations and datasets simultaneously with shared rate-limit management.
• Robust Error Handling: Integrated rate limiting, tenacity-based retries, and task-level timeouts.
• Detailed Logging: Automatic generation of raw API JSONL logs for every task, including request and response payloads.
• Extensible Architecture: Easy process for adding new model configurations or custom provider adapters.