THUSI-Lab/GameScaling

What it solves

Teaching an AI to play computer games directly from pixels. The hard part is turning a vision-language model into a real-time game controller: it must read what is on screen, decide what to do, and express that as precise keyboard and mouse commands — all without hand-written rules per game. GameScaling tackles this by pre-training a Qwen3.5 language model on actual gameplay so it learns how to react from vision alone.

How it works

GameScaling takes the current game frame (plus a short history of past frames and actions) and returns the next 200 ms of input: six key steps of 33 ms each, plus one mouse movement and scroll for the whole chunk. The models were continued pre-training on 1,600 hours of keyboard-and-mouse gameplay — no training code is included, only inference. A system prompt tells the model the game name, its rules, and the key bindings; the model can either act directly or "think" briefly before acting.

The repo provides the inference client (a Python class that builds requests and parses replies), the standard system prompt template with exactly three placeholders, a complete Doom example, and offline tests. Checkpoints are served with vLLM, downloaded from separate Hugging Face repositories — eight variants from 0.8B to 27B parameters, each in two training mixtures (S1: Minecraft only, S3: 27% Minecraft plus other games). The action format is a single string: mouse X/Y/scroll followed by six semicolon-separated key groups; the 74-key vocabulary maps to browser and Unreal Engine key names, and unparseable replies become no-ops.

Who it's for

Game AI researchers and hobbyists who want to run or study a model that plays games by watching the screen. It is also useful for anyone building an agent that must control keyboard-and-mouse environments, thanks to the well-defined action format and key mappings. Since there is no training code, this is aimed at people who want to use released models — adapt the system prompt to a new game, connect it to a game via the key mappings, and execute the returned actions.

Highlights

• Eight released checkpoints from 0.8B to 27B parameters (bfloat16); everything up to 9B runs on one GPU, 27B uses two GPUs with tensor parallelism
• Two training mixtures per size: S1 (Minecraft only) and S3 (27% Minecraft, rest from other games), all using 1,600 hours of gameplay
• A single standard system prompt with three placeholders (game name, rules, key bindings) — filling those is enough to try a new game
• Clear action format: one 200 ms chunk = six 33 ms key steps plus mouse movement and scroll; keys held across consecutive steps stay held instead of being re-pressed
• Two reasoning modes: think-then-act (model may reason before outputting the action) and act-only (reasoning closed as empty), both kept in the exact training format
• Robust parsing: malformed or truncated replies become explicit no-ops rather than crashing
• Offline unit tests run without a GPU, and the project also has a project page and a paper on arXiv (2610.28698)
• Code released under Apache 2.0