arcprize/arc-agi-benchmarking

Testing baseline LLMs performance across various models

What it solves

This project provides a standardized framework for benchmarking AI models on the ARC-AGI (Abstraction and Reasoning Corpus) tasks. It simplifies the process of running these complex reasoning tasks across various model providers, handling the operational overhead of API rate limits, retries, and performance scoring.

How it works

The system uses a set of model adapters to interface with different AI providers (such as OpenAI, Anthropic, Gemini, and Grok). Users define model configurations in a YAML file, specifying parameters like temperature and token limits. The benchmarking tool then executes tasks from the ARC-AGI datasets, manages concurrent requests across multiple configurations, and logs raw API interactions. Finally, a dedicated scoring script compares the model's generated outputs against the ground-truth solutions to calculate accuracy.

Who it’s for

It is designed for AI researchers and developers who want to evaluate the general intelligence and reasoning capabilities of LLMs using the ARC-AGI benchmark.

Highlights

  • Multi-Provider Support: Built-in adapters for a wide range of providers including OpenAI, Anthropic, Gemini, Fireworks, and Grok.
  • Robust Execution: Includes built-in rate limiting, tenacity-based retries, and asyncio for efficient batch processing.
  • Scalable Benchmarking: Ability to run multiple model configurations concurrently across different datasets with shared provider rate limits.
  • Detailed Logging: Automatic generation of raw API JSONL logs for every task to track model reasoning and request history.

Related

  • Project
    supermemoryai/memorybench

    A pluggable benchmarking framework for evaluating and comparing AI memory and context systems across different providers using standardized datasets and judge models.

  • Project
    harbor-framework/harbor

    A framework for evaluating and optimizing AI agents and language models by running benchmarks across scalable, containerized environments.

  • Project
    pathwaycom/arc-task-gen

    A task generator that creates new, distribution-matched ARC-AGI-1 style tasks to evaluate AI models on unseen problems and prevent benchmark contamination.

  • Project
    ServiceNow/AgentLab

    A framework for developing and evaluating web agents across various benchmarks, featuring large-scale parallel experimentation and trace visualization tools.

  • Project
    GraphRAG-Bench/GraphRAG-Benchmark

    A comprehensive benchmark for evaluating Graph Retrieval-Augmented Generation (GraphRAG) models to determine when they outperform traditional RAG across various reasoning and retrieval tasks.