StarSkirmish Bench v0.2: GPT-6 Astra Tops the LLM StarCraft Bot Benchmark

StarSkirmish Bench v0.2 ranks nine frontier LLMs by how strong a StarCraft: Brood War bot each can write in one hour of wall-clock time, and GPT-6 Astra wins with a score of 57 out of 100. The headline result is not that a model leads, but that the best model still beats the strongest human-written bots in only 7% of its games — the benchmark shows no sign of saturating.

What StarSkirmish Bench v0.2 measures

StarSkirmish is a benchmark where LLMs write bots for StarCraft: Brood War. Each LLM gets one hour to write a Protoss bot in C++, which is then evaluated against other LLMs' bots and established human-written bots. The benchmark tests long-horizon reasoning and agentic coding, because models must incorporate learnings from practice games to improve their bot within the hour.

v0.2 updates StarSkirmish Bench v0.1 with flagship models from OpenAI, Anthropic, Google and Meta, each running in its lab's first-party CLI harness. An LLM's score reflects its win rate against a roster of competitive human-written bots and demo bots.

Leaderboard: GPT-6 Astra leads, Gemini 4 Argon and Claude Opus 5.5 tie

Scores are average win rates (0–100) of each model's 5 runs against a roster of competitive human-written bots, from one-hour runs in each lab's first-party CLI harness.

Model	StarSkirmish Bench v0.2 score
GPT-6 Astra	57
GPT-6.1 Sol	46
Gemini 4 Argon	39
Claude Opus 5.5	39
Claude Fable 5.1	37
Claude Sonnet 5.5	26
Muse Wonder Full	21
Claude Haiku 5.5	21
GPT-6 Luna	13

GPT-6 Astra is the top-scoring model in StarSkirmish Bench v0.2, a clear step ahead of GPT-6.1 Sol in second place. Gemini 4 Argon, Claude Opus 5.5 and Claude Fable 5.1 are functionally tied behind them, all scoring between 37 and 39.

One caveat on reasoning effort: Gemini 4 Argon runs at high, the highest reasoning effort available to it, while every other model runs at max.

Cost: GPT-6.1 Sol offers the best value per run

StarSkirmish Bench score plotted against the average API cost of one one-hour run (cost on a log scale) puts GPT-6.1 Sol in the most attractive quadrant. GPT-6.1 Sol is the only LLM in that quadrant, scoring second at just over a tenth of GPT-6 Astra's cost per run. Muse Wonder Full's pricing is based on Muse Wonder's API pricing.

No signs of saturating: human-written bots still dominate

The strongest human-written bots are still well ahead of every model. Against Stardust, PurpleWave and BananaBrain, the best model, GPT-6 Astra, wins only 7% of its games.

Average win rate of each model's 5 bots against the top human-written bots Stardust, PurpleWave and BananaBrain:

• GPT-6 Astra: 7
• GPT-6.1 Sol: 4
• Gemini 4 Argon: 2
• Claude Opus 5.5: 2
• Claude Fable 5.1: 2
• Claude Sonnet 5.5: <1
• Muse Wonder Full: <1
• Claude Haiku 5.5: <1
• GPT-6 Luna: 0

One-hour reasoning limits are a factor, but these matches are played on three two-player maps. Real human-written bot tournament conditions are significantly harder.

Gemini 4 Argon: one-shot coding, Reavers, and a timing attack

Gemini 4 Argon places 3rd in v0.2, replacing v0.1's Gemini 3.8 Flash. Its distinctive behavior is that it uniquely preferred writing bots via one-shots: its five runs had the fewest builds and code versions of all the traced runs.

41% of Gemini 4 Argon's practice games were against high-tier opponents, second only to GPT-6 Astra, yet its bots win only 7% of their games against those tiers, compared with Astra's 28%. It averages 2.5 upgrades a game, the most in the field.

Gemini 4 Argon's favorite strategy is a timing attack — Dragoons and Zealots held at home behind cannons for one big push. Its favorite unit is the Reaver, and its second-best bot builds Reavers in over 70% of its games. In one recorded clip, Argon's one big push meets an equal army at 192 supply, and Psionic Storms win the trade.

Muse Wonder Full: Zealot floods and heavy Observer use

Muse Wonder Full places 7th in v0.2, replacing v0.1's Muse Spark 1.3. Its favorite strategy is macro: a fast expand into 5 to 9 Gateways, attacking only once it outnumbers the army it has seen. It builds 3 Observers per game, the most of any model and 1.6x the field, and 22 Zealots per game — as in v0.1, a Zealot-heavy army. In one clip, Muse's Zealot flood runs down a Dragoon army.

Muse Wonder Full is also one of the models that rarely or never practices against the strongest bots on the roster.

The strongest predictor of score is practicing against strong bots

Each run's agent trace records how the model spent its hour, and the strongest predictor of a model's score is how much it practiced against the 7 strongest bots on the roster. GPT-6 Astra, GPT-6.1 Sol and Gemini 4 Argon start playing these bots early in the hour, while their own bots still lose to them. Muse Wonder Full, Claude Haiku 5.5 and GPT-6 Luna rarely or never do.

Claude Fable 5.1 is the outlier: it scores 37 while practicing almost entirely against weaker bots.

Run-to-run variance is large, especially for Gemini 4 Argon

Win rates vary between runs of the same model. Each model's 5 runs produce a spread, with the mean equal to the model's Bench score:

Model	Score	σ
GPT-6 Astra	57	3.4
GPT-6.1 Sol	46	6.0
Gemini 4 Argon	39	7.0
Claude Opus 5.5	39	3.2
Claude Fable 5.1	37	2.9
Claude Sonnet 5.5	26	8.4
Muse Wonder Full	21	3.0
Claude Haiku 5.5	21	6.1
GPT-6 Luna	13	5.6

Gemini 4 Argon varies far more than the other labs' flagships: its best and worst runs are 18 points apart, against 9 for GPT-6 Astra and 8 for Claude Opus 5.5.

How the benchmark works: objective, tools and tournament setup

Objective

Each LLM writes a Protoss bot in C++ against BWAPI 4.4.0, played on OpenBW. Games are Protoss versus Protoss on three ladder maps: Heartbreak Ridge, Benzene and Destination. A game ends when one side's buildings are all destroyed. Games that reach the 60-minute cap (86,400 frames) are decided as on the BASIL ladder: the higher kills + razings score wins.

The LLM starts from a skeleton bot that only mines, with the source code of the three demo bots and two standard bot libraries: FAP, a combat simulator, and BWEM, a map analyzer. Whatever is in its bot folder when the hour ends is its final bot, so it has to keep the bot compiling.

The LLM has tools to compile its bot, play practice games, and read each game's summary, transcript and its bot's own log. Practice opponents are grouped into tiers: D (the three demo bots), C and B (three mid-ranked human-written bots each), A (BananaBrain, Locutus) and S (Stardust, PurpleWave).

Tournament setup

The v0.2 tournament ran 45 LLM bots against 13 opponents for 10,530 games, 18 games per pair. Each model's 5 final bots play each of the 13 roster bots — the 10 competitive human-written bots and the 3 demo bots — 6 times on each map, on fresh seeds. Unlike v0.1, the LLM bots don't play each other.

Harness and reasoning effort

StarSkirmish Bench v0.1 ran every model in the same neutral third-party harness at high reasoning. v0.2 runs each model in its lab's first-party CLI harness at the highest reasoning setting it offers:

• Anthropic (Claude models): Claude Code, version 2.1.296
• OpenAI (GPT models): Codex, version 0.162.1
• Google (Gemini models): Antigravity CLI, version 1.3.5
• Meta (Muse models): Muse Code, version 1.4.7

Every v0.2 harness compacts the model's context at about 250k tokens. When a conversation nears that point, the CLI condenses it and the model carries on. Codex, for example, gives the GPT models its default 272k token window and triggers compression once 90% filled.

The CLIs' web search and web fetch tools are turned off, and the agent has no other internet access. The CLI is installed before the clock starts, and the hour counts from when the agent starts. There is no submit tool: the harness collects the bot when the time runs out.

Scoring

v0.2 replaces v0.1's Elo ratings with plain win rates. A bot's score is its win rate over its 234 tournament games, and a model's StarSkirmish Bench score is the average of its 5 bots' scores, from 0 (every game lost) to 100 (every game won).

Because every bot plays the same fixed roster, its games don't depend on the other LLM bots in the field. They all run in parallel, and a new model's bots can be scored without replaying anyone else's. A fixed roster also keeps the bar the same over time, so scores from different model generations are more directly comparable than Elo ratings fitted within one field.

Correlations: StarSkirmish tracks Vals IOI and WeirdML, not the popular evals

StarSkirmish no longer tracks the most widely reported evals. It barely moves with the Artificial Analysis Intelligence Index (ρ 0.26), where GPT-6 Astra, the top model here, only ties for 4th of these 9 models. It has also drifted away from Terminal-Bench 4.0, which it tracked closely in v0.1. Claude Sonnet 5.5 and Muse Wonder Full both rank near the top of Terminal-Bench 4.0 but near the bottom here.

StarSkirmish still tracks Vals IOI (ρ 0.95), olympiad programming problems solved in C++, and WeirdML v3 (ρ 1.00, n = 7; Gemini 4 Argon and Muse Wonder Full had not been included at time of writing), an independent benchmark of unusual machine learning coding tasks. One reading offered by the benchmark authors is that the most heavily benchmarked evals have become targets that models are tuned for, while less targeted tests of long-horizon, difficult coding tasks still move together.

Discussion: what the benchmark actually measures

The Hacker News thread on StarSkirmish Bench v0.2 (150 points, 69 comments) focused less on the leaderboard and more on whether the benchmark isolates a meaningful capability.

One recurring objection is that the score conflates several skills. A commenter wrote:

I am not sure I like this benchmark, but mostly, because I am not sure what it is really evaluating... multiple factors seem to complicate what it is supposed to look at (just top of my head): code generation speed, logic analysis, understanding of the game platform and how use of C may influence it, considered approaches, tactics and strategies.

Another asked for qualitative analysis rather than win rates:

Is there anything interesting to learn from the agent written bots? Or are they just inferior versions of the best bots because the time limit is so short? Is there any indication that they're trained on the code from the human written bots... How much are the bots on rails applying the same techniques every time?

A related concern is pre-training contamination, since StarCraft bot code is public. One commenter noted the human-written bots still win, which is evidence against trivial memorization but not a full explanation. Another framed the benchmark as "a version of how well they can improve upon those solutions public out there."

Several commenters proposed variants. One suggested benchmarking against Pluto, described as a neural-network bot that can take games off the best Brood War players. Others asked what happens if agents play StarCraft directly instead of writing bots — pointing to broodbench.com, a separate project where LLMs play StarCraft directly. One commenter wanted the inverse experiment: "how long does each model take to get >50% WR against bots of various tiers." Another asked how the best human players would fare against the LLM-written bots, and one asked why the benchmark uses Protoss.

The most-upvoted technical point was about harness effects. Quoting the benchmark's own description of first-party CLIs and disabled web access, a commenter wrote:

One thing we're learning is that harness is HUGE. The difference in performance for the same underlying model can change dramatically across: 1. harnesses 2. allowed tools (no web search/fetch - how does the model refer to documentation?) 3. thinking levels

A dissenting data point came from a developer running bots on a ModernUO server, who reported the opposite ranking in their own work:

I've been populating a ModernUO server with bots, and have found Opus 5.5 to be substantially better at understanding and building it. I actually started with 6.1 Sol and 6 Astra but pulled them after a week of little progress, had Opus 5.5 take over and it rebuilt everything properly in a night and has carried it forward quite a way since.

One commenter noted that the relative ordering in StarSkirmish Bench v0.2 (GPT-6 Astra > GPT-6.1 Sol > Gemini 4 Argon > Claude Opus 5.5 > Claude Fable 5.1 > Claude Sonnet 5.5) matches the ordering of an independent 9x9 Go benchmark that uses KataGo opponents as Elo anchors, despite the two benchmarks testing different problems.

Smaller critiques included a note that the cost chart's "Pareto curve" is not a true Pareto curve because Claude Haiku 5.5 sits inside it rather than on it, a question about whether the video clips show a GLM model that does not appear in any chart, and a joke that the benchmark omits actions per minute, "clearly the best indicator of performance." Commenters also pointed to aiarena.net for StarCraft II bot competition and suggested extending the format to StarCraft II, Red Alert 2, Counter-Strike and Cities: Skylines.