Hugging Face Open Chain of Thought Leaderboard

Hugging Face has launched the Open Chain of Thought (CoT) Leaderboard, a specialized evaluation framework designed to measure how much chain-of-thought prompting actually improves a model's accuracy on complex reasoning tasks. Unlike traditional leaderboards that track absolute accuracy, this leaderboard focuses on the marginal gain provided by the reasoning process itself.

Measuring the Impact of Reasoning via Accuracy Gain

The Open CoT Leaderboard evaluates the effectiveness of a model's reasoning traces by calculating the difference between its performance with and without CoT prompting. This approach is designed to isolate the impact of the reasoning process from the model's baseline knowledge.

The Accuracy Gain Formula: accuracy gain Δ = accuracy with CoT – accuracy w/o CoT

To establish the baseline (accuracy without CoT), the leaderboard uses loglikelihood accuracy for multiple-choice evaluations. This methodology aims to be more robust against training data contamination; while a model might have seen a specific answer during training, it may still struggle to generate a valid reasoning path to reach that answer.

Evaluation Tasks and Benchmarks

The leaderboard utilizes tasks that require common-sense knowledge and generic reasoning, ensuring they remain challenging even for high-performing LLMs. All tasks are presented as multiple-choice problems.

The evaluation suite includes:

  • LogiQA: Including a new translation of the original version and version 2.0.
  • LSAT: Subsets focusing on analytical reasoning, logical reasoning, and reading comprehension.

Most of these tasks are sourced from the AGIEval benchmark and re-published via logikon-bench.

CoT Generation Regimes and Prompting Strategies

To account for the wide variety of CoT prompting techniques, the leaderboard employs modular prompt chains and tests models across six different "generation regimes" (combinations of two prompting strategies and three decoding parameters).

Prompting Strategies

  1. Classic: The problem is presented followed by the instruction "Let’s think step by step."
  2. Reflect: The model is instructed to reflect on the problem in general terms before proceeding with a step-by-step solution.

Decoding Parameters

  • Greedy decoding
  • Beam search (n=2)
  • Sampling (T=.3)

For every example in the test datasets, the model generates one trace for each of the six regimes. The leaderboard reports the best marginal accuracy gain achieved under any of these regimes for each model/task pair.

Key Findings and Insights

Initial results from an evaluation of 30 models reveal several trends regarding the reasoning capabilities of open LLMs:

  • Effectiveness of Small Models: Relatively small models (e.g., 7B parameters) can exhibit effective CoT reasoning. In some instances, smaller models like Phi-2 show greater accuracy gains from CoT traces than larger models like Mixtral.
  • Impact of Fine-tuning: Instruction- and chat-finetuning significantly improves performance. Finetuning enhances both the baseline accuracy (without CoT) and the marginal gains achieved when CoT is applied.
  • Inconsistency of CoT Effects: There is no universally superior CoT generation regime. The effectiveness of a strategy varies by model and task, and in some cases, CoT can actually reduce accuracy, indicating that creating a robust and reliable implementation of CoT remains a challenge.

Future Development and Contributions

Hugging Face is seeking community contributions to expand the leaderboard in several areas:

  • Model Submissions: Open LLMs can be submitted for evaluation via the Submission tab on the leaderboard's HF space.
  • Analytical Tools: Development of an Open CoT Dashboard to visualize baseline accuracies, variance, and reasoning trace properties (such as length).
  • Expanded Strategies: Implementation of additional CoT regimes such as tree-of-thought, self-consistency, self-check, or debating.
  • Dataset Expansion: Inclusion of more challenging reasoning tasks as compute resources allow.

Sources