GamePhanesStudio/GamePhanes

What it solves

Game Terminal-Bench provides a specialized benchmark for evaluating coding agents. Unlike standard terminal benchmarks that only check files and exit codes, this project addresses the need to evaluate agents on their ability to build, debug, and repair interactive software (like games) where the "truth" is determined by the actual runtime behavior and state changes of the application.

How it works

The benchmark uses a loop of instruction, inspection, editing, running, and observation to test agents. It is built on the Harbor task contract, meaning each task is a self-contained directory containing a Dockerfile for the environment, a starter project (often broken or incomplete), instructions for the agent, and a set of tests/probes to verify the runtime behavior from the outside.

Who it’s for

This tool is designed for researchers and developers building coding agents that need to be tested against complex, interactive software engineering tasks rather than simple script-writing tasks.

Highlights

• Runtime-focused evaluation: Judges agents on whether the software actually runs and behaves as intended, not just on code changes.
• Harbor-compatible: Uses a standardized task format for portable and reproducible evaluation runs.
• Interactive targets: Focuses on game development and interactive applications to provide a high-difficulty ceiling for agent debugging.
• Deterministic verification: Employs evaluator-controlled probes to obtain evidence of success without relying on GUI screenshots.