Brood War Bench: LLMs Play StarCraft II – Codex Astra Leads, All Models Beginner‑Level

Takeaway

Codex Astra (xhigh) achieved a perfect 100 % win‑rate, but every LLM in the benchmark behaved only at a beginner level, unable to execute sustained strategies or defend against simple attacks.

Benchmark Overview

The "Brood War Bench" benchmark pits large language models (LLMs) against each other in a custom StarCraft II: Brood War environment where agents issue game commands directly. Each model is run at several effort settings (low, medium, xhigh) to measure win‑rate, actions‑per‑minute (APM), and monetary cost per game.

Leaderboard Summary

Rank Model / Effort Wins Losses APM Cost / game Win rate
🥇 Codex Astra / xhigh 18 0 12.6 $10.54 100 %
🥈 Codex Astra / medium 16 2 17.2 $15.11 88.9 %
🥉 Claude Fable 15 3 12.6 $12.24 83.3 %
4 Codex Astra / low 14 4 25.7 $21.07 77.8 %
5 Codex 5.6 Sol / medium 13 5 10.1 $5.12 72.2 %
… … … … … … …

The top three entries are all from OpenAI’s Codex family, with Claude Fable the only non‑Codex model in the top five.

Model Behaviors

Codex Astra – Aggressive Disruption, Weak Macro

"Codex's strongest recurring idea was disruption. It often sent a Probe across the map to attack workers or buildings, forcing opponents to waste time thinking." – Ben Swerdlov

  • Cheese tactics – Early probes and single‑unit raids repeatedly caught opponents off‑guard.
  • Fragmented sub‑agents – Separate agents handled economy, production, and combat without coordination, leading to piecemeal attacks.
  • Beginner mistake – Units were launched one at a time instead of waiting for a critical mass, limiting army effectiveness.
  • Persistence – In a loss, Codex 5.6 Terra lifted its Command Center and survived six extra minutes, showing an ability to avoid immediate defeat.

Claude Fable – Ambitious but Unfinished

"Fable usually tried to build an economy and climb the tech tree instead of stopping at the first unit available." – Ben Swerdlov

  • Long‑term planning – Built advanced tech (Lair, Spire, Mutalisks; Robotics Facility, Templar Archives) and sometimes won.
  • Execution gaps – Even with strong tech, Fable often failed to translate upgrades into a decisive army (e.g., reached a Factory and Academy but was overrun by Opus 5).

Grok 4.6 – Too Much Thinking, Too Little Acting

"Grok 4.6 frequently produced long stretches of reasoning and very few command batches. In one 43‑minute game it issued only six command batches." – Ben Swerdlov

  • Low APM – Averaged 11 APM, rarely fielded combat units.
  • Turn‑based illusion – The model behaved as if the game were paused for reasoning, leading to no meaningful combat.

Quantitative Findings

Metric Codex Astra (xhigh) Claude Fable Grok 4.6
Average APM 12.6 12.6 11.1
Average army size (units) 8.3 7.6 2.0
Average structures 6.2 7.4 4.5
Unspent minerals 240.6 248.6 536.6
Win rate 100 % 83.3 % 0 %

All models spent a substantial portion of resources idle, especially Grok, which left over half a million minerals unspent on average.

Head‑to‑Head Matrix Insights

The full 19 × 19 matrix (link in the source) shows that:

  • Codex Astra (xhigh) defeats every other configuration.
  • Even the strongest non‑Codex model (Claude Fable) loses to Codex Astra at both effort levels.
  • Grok never wins a match; its best result is a time‑limit draw.
  • Claude Opus 5 and Claude Sonnet achieve modest win rates only against lower‑effort Codex variants.

Community Reactions

"Back in 2010, during the early days of bwapi, there was a Brood War AI tournament… It’s interesting to see how different the approaches were back then, vs this or DeepMind’s SC2 work." – @AntiRush

"Good example of how these things sometimes spend way too much time thinking to be useful" – @xyzsparetimexyz

"I would love to create one of these benchmarks for Age of Empires 2, but I have no idea how to make the AIs play it." – @windowshopping

"Astra had an explicit medium/xhigh levels, Fable – just Fable. What reasoning level was used?" – @leobuskin

The comments highlight nostalgia for classic StarCraft AI tournaments, curiosity about the real‑time vs turn‑based nature of the harness, and suggestions for extending the benchmark to other RTS or strategy games.

How the Benchmark Was Run

  • A round‑robin matrix paired every model‑effort configuration against every other.
  • Matches were executed in parallel on Freestyle VMs.
  • Both the game engine state and the agents' logs were recorded for each game.
  • Costs were calculated using token‑based pricing for each model (e.g., OpenAI token rates, Claude token rates).

Implications and Future Directions

  • Real‑time reasoning remains a bottleneck. Even the best‑performing LLM (Codex Astra) resorts to cheap disruption tactics rather than sophisticated macro management.
  • Cost‑performance trade‑offs are visible. Higher effort levels increase win‑rate but also raise per‑game cost; some lower‑effort settings (e.g., Codex 5.6 Sol / low) achieve respectable win rates at a fraction of the cost.
  • Benchmark extensibility. Community interest suggests applying the same framework to other RTS titles (Age of Empires II, StarCraft II Remastered) or to turn‑based strategy games like Go (see GoBench).
  • Model architecture matters. The split‑sub‑agent pattern observed in Codex hints at a possible architectural improvement: a shared state or coordination layer could mitigate the “one‑unit‑at‑a‑time” weakness.

Play Your Own Match

The benchmark is publicly accessible. Users can bring their own agents and compete against the published models:

Play Brood War

Sources

Related