fstandhartinger/jevbench

What it solves

JevBench provides a standardized way to evaluate and rank AI decision models. It addresses the need for consistent performance measurement across different systems by assessing not just raw intelligence, but also how well a model understands its own confidence, how quickly it responds, and how much it costs to run.

How it works

JevBench evaluates models using a set of frozen, cross-reviewed decision tasks. It measures four key axes: Intelligence (accuracy corrected for chance), Calibration (how well the model's confidence matches its actual accuracy), Speed (latency), and Cost (price per 1,000 decisions). These metrics are combined into a composite score using a weighted harmonic mean. The benchmark uses both public and held-out (private) tasks to ensure models are tested on data they haven't been specifically trained to solve.

Who it’s for

This project is for developers, researchers, and AI practitioners who need to compare the performance, efficiency, and reliability of different AI decision models. It is particularly useful for those working with Jev-compatible systems or looking for data-driven insights into model tradeoffs.

Highlights

• Evaluates models on four distinct axes: Intelligence, Calibration, Speed, and Cost.
• Uses a composite scoring method that penalizes models that fall below specific performance floors.
• Maintains a transparent leaderboard with versioned methodology to ensure historical comparability.
• Distinguishes between raw reasoning ability and production-ready metrics like latency and cost.
• Provides clear data cards and manifest files for reproducible results.