haowang02/codex-candy-eval

Codex 降智测试

What it solves

This tool helps users evaluate the performance and reasoning consistency of Codex models by testing them against a specific math problem.

How it works

The script uses the local Codex CLI to run a batch of tests on a "candy math problem" where the correct answer is 21. It automatically tracks the number of reasoning tokens used and calculates the success rate by checking if the number 21 appears in the model's output.

Who it’s for

It is designed for users who have the Codex CLI installed and want to benchmark different model versions or reasoning effort settings.

Highlights

  • Zero third-party dependencies required.
  • Supports customizable model selection and reasoning effort levels (low, medium, high, xhigh).
  • Allows for multiple test iterations to gather statistical data.
  • Provides simple one-line execution commands via wget or curl.

Related

  • Project
    zilliztech/VectorDBBench

    VectorDBBench is a benchmarking tool that allows users to compare the performance and cost-effectiveness of various open-source and cloud-based vector databases using standardized, repeatable workloads.

  • Project
    hijohnnylin/neuronpedia

    An open-source interpretability platform for exploring and analyzing neural network features, providing tools for automated interpretation, steering, and circuit visualization.

  • Project
    kornelski/dssim

    A tool that measures the visual similarity between images using an algorithm that mimics human perception, helping users benchmark image compression quality.

  • Project
    JackHopkins/factorio-learning-environment

    An open-source framework for developing and evaluating LLM agents in Factorio, using a code synthesis REPL pattern to test long-term planning and resource management.