GamePhanesStudio/GamePhanes

What it solves

GameForgeBench addresses the limitation of existing coding benchmarks that only check for file changes or exit codes. It provides a way to evaluate AI coding agents on their ability to actually build, debug, and repair interactive software—specifically games—where the "truth" is determined by whether the project launches and behaves correctly within a real game engine runtime.

How it works

The benchmark uses a loop of instruction, inspection, editing, and runtime observation. It provides agents with a starter project and a natural-language request. To pass, the agent must modify the code such that a deterministic native probe can verify the intended behavior during execution.

Key technical components include:

• Hermetic Environments: Each task uses Docker to ensure a reproducible environment with pinned engines and dependencies.
• Multi-layer Evidence: Evaluation is split into artifact checks (files), engine checks (startup/parsing), and behavior checks (state transitions and events).
• Binary Scoring: A task is marked as passed (1) only if all required behavioral checks pass; otherwise, it is a fail (0).

Who it’s for

It is designed for researchers and developers building AI coding agents that need to be tested on complex, stateful engineering tasks across various game engines (such as Godot, Unity, Minecraft, Roblox, and C++).

Highlights

• Runtime-Based Verification: Rewards actual functional behavior in-engine rather than just code similarity or file presence.
• Broad Engine Support: Includes tasks for Godot, Unity, Roblox, Minecraft, Unreal, and C++.
• Deterministic Probes: Uses native probes to avoid the unpredictability of LLM-based judges in the scoring path.
• Comprehensive Task Format: Every task includes a starter project, instructions, a Dockerized environment, and a reference solution.