Jevman: A Pac-Man Benchmark for AI Decision Models

Jevman evaluates AI decision-making via Pac-Man

Jevman is an open-source benchmark designed to test the real-time decision-making capabilities of AI models. By requiring models to play Pac-Man against classic arcade ghosts, the benchmark measures a model's ability to process game state and execute strategic moves under strict latency constraints.

Performance Leaderboard and Model Rankings

As of October 7, 2026, the benchmark tested six models over 100 games each. Three models are currently tied for the top spot based on their mean scores within a 95% margin of error:

jev 1.13 (TypeSafe): Mean score of 2,750 ± 218, with a high score of 6,380 and the lowest backup move rate (3.4%).
GPT-6 Luna (OpenAI): Mean score of 2,568 ± 150, with a high score of 5,920.
Clef Flash (Cloudflare): Mean score of 2,538 ± 120, with a high score of 4,260.

Other models, including Cloudflare's Clef, Jared Palmer's Kev 4B, and Convai Innovations' Laya, follow in the rankings. Laya recorded the lowest latency (104 ms) but the lowest mean score (639 ± 24).

Benchmark Methodology

The Jevman benchmark employs a specific set of rules to ensure consistency and consistency in testing:

Game State and Decision Process

At every junction in the maze, the game sends the current state—including the positions of pellets and ghosts—as a JSON object to the model's HTTP endpoint. The model must return a probability for each possible direction. Pac-Man then selects a direction based on these probabilities.

Constraints and Scoring

Game Duration: Games are capped at 5 minutes, though the longest recorded run lasted 2 minutes and 24 seconds.
Latency Deadline: The model must provide an answer within 2 seconds. If the model exceeds this limit, a simple backup rule is used to determine the move, which is recorded as a "backup move."
Ranking Metric: Models are ranked by mean score with a 95% margin of error (±2 standard errors). Models whose scores fall within each other's margin are considered tied.

Open Source Contribution and Integration

Jevman is open source, allowing any model behind an HTTP endpoint to be benchmarked. Developers can integrate their models by putting them behind an endpoint, running the benchmark locally, and submitting a pull request. The project's CI system replays every submitted game to verify the scores before adding the model to the leaderboard.

Community Insights and Technical Critiques

Community discussion on Hacker News highlighted several technical considerations regarding the benchmark's architecture:

"Running cloud API inference for Pac-Man is peak modern software engineering. A 2 MB local ONNX model on a CPU core beats network round-trip jitter every time."

Critics pointed out that the model's decision-making occurs only at junctions rather than at every game tick, which may limit the model's ability to change direction dynamically. Others suggested that including a standard Large Language Model (LLM) as a baseline for comparison would provide better context for the performance of specialized decision models.