Pac-Bench: Evaluating LLM One-Shot Game Development Capabilities

Pac-Bench reveals a significant performance gap in one-shot game implementation

Pac-Bench is a benchmark designed to test how well frontier Large Language Models (LLMs) can generate a fully functional, arcade-accurate Pac-Man clone in a single shot. The results demonstrate a wide spectrum of capability, ranging from nearly perfect replicas to complete syntax errors.

Claude Opus 5.5 emerged as the top performer, scoring 99/100, followed closely by Claude Fable 5.1 (96/100) and Claude Sonnet 5.5 (95/100). In contrast, models like Minimax-m3 and Muse-Glimmer-30b failed significantly, scoring 2/100 and 4/100 respectively.

Performance Breakdown by Model

Top Tier: High Fidelity Replicas

Models in the 90+ range successfully implemented core mechanics, including arcade-accurate ghost AI, collision detection, and complex audio systems.

  • Claude Opus 5.5 (99/100): Achieved nearly perfect marks. It implemented a full two-phrase intro with a bass line, a continuous siren that rises with progress, and arcade-style death sounds.
  • Claude Fable 5.1 (96/100): Highly accurate but lacked a separate power-pellet sound and used a generic arpeggio for the intro.
  • Claude Sonnet 5.5 (95/100): Implemented a wide array of controls (arrows, WASD, swipe) but produced a "thinner" sound profile with separate beeps instead of a smooth siren sweep.
  • Grok-4.7 (94/100): Successfully implemented the maze and ghost AI but failed on specific control interactions, such as ignoring arrow keys on the title screen.

Mid Tier: Functional but Flawed

Models scoring between 60 and 90 generally produced playable games but with noticeable mechanical or visual bugs.

  • GPT-5.6-sol (90/100): Lacked a siren and mute function, and controls only worked at tile centers.
  • GLM-5.3-flash (88/100): Created a smaller 19x21 maze and had potential audio blocking issues in Safari.
  • GPT-6-astra (87/100): Featured a bug where revived ghosts could be eaten again during the same power-up cycle.
  • DeepSeek-v4.1-flash (72/100): Failed entirely on ghost AI, as frightened ghosts continued to chase Pac-Man.

Low Tier: Non-Functional Implementations

Models scoring below 60 often struggled with basic spatial logic, resulting in "dead ends" in the maze or ghosts that froze upon spawning.

  • Claude Opus 5 (58/100): Suffered from a drawing offset bug where Pac-Man and ghosts appeared inside walls.
  • Gemini-3.7-flash (38/100): Produced a game where Pac-Man could leave the maze via an off-screen column.
  • Grok-4.5 (20/100): Generated a maze with 23 dead ends and 10 unreachable pellets, making the level unwinnable.
  • Minimax-m3 (2/100): Failed with a syntax error resulting in a blank canvas.

Scoring Methodology

The benchmark evaluates models based on five primary technical criteria:

  1. Controls (20 pts): Support for various input methods (keyboard, swipe, click) and pause functionality.
  2. Ghosts (25 pts): Accuracy of ghost AI patterns and the "eyes" return animation.
  3. Pac-Man Movement (20 pts): Ensuring the character does not get stuck in walls.
  4. Maze Integrity (20 pts): Absence of dead ends and unreachable pellets.
  5. Sound (15 pts): Implementation of the intro jingle, siren, and death sounds.

Community Analysis and Counterpoints

Discussion among developers and researchers suggests that the high scores of top models may be attributed to the prevalence of Pac-Man clones in training data rather than true reasoning.

"Clearly some new RLAAS/dataset/env is being used for this now... Everything is going to go from 0->1 on this benchmark in short order because of that."

Other critics argued that the prompt used was too vague, suggesting that the benchmark tests a model's ability to "fill in missing context" rather than its ability to follow strict technical specifications. Some developers noted that while the games look like Pac-Man, they often lack the 1:1 behavioral replication of the original arcade ghost patterns that a manual implementation would require.

"LLM implementations lose lots of details... the behavior of the ghost is not the same as the original. If I have not implemented the game myself, I can not tell the differences."

Finally, some users expressed concern that the ability of AI to one-shot complex clones could demotivate new developers from learning the fundamentals of game programming by bypassing the "pleasure of figuring it out."

Sources

Related

  • Dispatch
    Benchmarking Nine Coding Harnesses on a MacBook Pro with Local Qwen 3.8 27B Model

    A systematic benchmark of nine coding agent harnesses on a M4 MacBook Pro shows that lean harnesses like pi, mini‑swe‑agent, and chad achieve 8 tokens/s, while heavyweight harnesses such as opencode and crush suffer multi‑minute startup delays and drop to 5–6 tokens/s.

  • Dispatch
    BigCodeBench: A New Benchmark for Complex Python Code Generation

    Hugging Face has released BigCodeBench, a benchmark of 1,140 function-level tasks designed to evaluate LLMs on practical programming and diverse library usage, addressing the simplicity and contamination issues of HumanEval.

  • Dispatch
    Claude Fable 5 vs GPT-5.6 Sol $100 AI Music Video Build‑off

    Claude Fable 5 and GPT‑5.6 Sol each produced a full‑length music video for Uptown Funk on $25 and $100 budgets, revealing distinct tool‑use patterns, cost structures, and creative shortcomings.

  • Project
    MatthewCYM/VoiceBench

    VoiceBench is a benchmark suite for evaluating LLM‑based voice assistants. It supplies a Hugging‑Face dataset with audio‑driven QA and instruction‑following tasks, a command‑line toolkit to run a model, collect spoken responses, and automatically score them (via GPT‑4o‑mini for open‑ended sets). Results are posted on a public leaderboard, and the repo includes a curated list of recent voice‑assistant papers.

  • Dispatch
    chess-postmortem-skills: AI-Powered Chess Game Analysis for Claude Code

    The chess-postmortem-skills project introduces Claude Code skills that combine Stockfish engine analysis with LLM-generated natural language explanations and narrated videos to provide deep, personalized chess post-mortems.