Brood War Bench: Evaluating LLM Strategic Capability in Real-Time Strategy

LLMs Struggle with Real-Time Strategy Orchestration

No current large language model (LLM) can play StarCraft: Brood War beyond a beginner level, according to the Brood War Bench benchmark. While some models demonstrate basic tactical disruption, they consistently fail at complex army composition, defending simple attacks, and maintaining the sustained production (macro-management) required for competitive play.

Performance Leaderboard and Key Findings

Codex Astra emerged as the clear leader in the benchmark, achieving a 100% win rate in its "xhigh" effort setting. However, this dominance is relative to other LLMs rather than a measure of professional-grade skill.

Rank	System	Wins	Losses	Win Rate	Cost / Game
1	Codex Astra / xhigh	18	0	100.0%	$10.54
2	Codex Astra / medium	16	2	88.9%	$15.11
3	Claude Fable	15	3	83.3%	$12.24
4	Codex Astra / low	14	4	77.8%	$21.07
5	Codex 5.6 Sol / medium	13	5	72.2%	$5.12

The "Cost of Thinking" in Real-Time Environments

A critical observation from the benchmark is the tension between reasoning depth and real-time execution. Older models frequently treated the real-time strategy (RTS) game as a turn-based experience, leading to their destruction while they were processing their next move.

Newer models are more cognizant of the time cost of thinking, but this still creates a performance ceiling. For example, Grok 4.6 demonstrated a severe failure in this area: in one 43-minute game, it generated 11,138 reasoning tokens but issued only six command batches, failing to field a single combat unit.

Model-Specific Behavioral Analysis

Codex: Tactical "Cheese" over Macro-Management

Codex models prioritized disruption over sustainable growth. A recurring successful strategy was sending a single Probe across the map to attack enemy workers or buildings. This "cheese" strategy worked because opposing agents often spent significant time reasoning about how to respond to a single unit rather than continuing their own production.

However, Codex struggled with sustained production, often delaying technology and trickling units into attacks one by one rather than building a critical mass. To manage this, Codex often created separate subagents for economy, production, and control, but these subagents lacked communication, resulting in uncoordinated attacks.

Claude Fable: Ambition and Tech Climbing

Claude Fable demonstrated a more "earnest" approach to the game, attempting to build a functioning economy and climb the tech tree. Fable successfully reached advanced units like Mutalisks and utilized structures such as the Robotics Facility and Templar Archives. Despite this ambition, Fable still struggled with the execution required to turn a high-tech build into a winning army.

Benchmark Methodology

The benchmark utilized a round-robin matrix where various model and effort configurations played against each other in parallel across Freestyle VMs. The harness recorded game-engine data and agent logs for every match.

Resource and Production Metrics

Across the tested models, the average game state at the 5-minute mark highlights the disparity in capability:

• Workers: Claude Fable averaged 16.7 workers, compared to Grok 4.6's 7.4.
• Army Size: Codex Astra averaged 8.3 units, while Grok 4.6 averaged 2.
• Supply Used: Codex Astra and Claude Fable both hovered around 26.5 supply, whereas Grok 4.6 used only 11.3.

Community Insights and Counterpoints

Discussion among the technical community emphasizes that this benchmark tests long-term strategic thinking and orchestration rather than just tactical solutions.

"Something all the benchmarks seem to be missing is strategy... here requires actual long term thinking and tactical thinking, balancing and orchestration."

Other contributors noted the difference between these general-purpose LLMs and specialized AI. While LLMs struggle, RL-trained bots (such as Pluto) are currently capable of defeating professional players on the Korean ladder. This suggests that the gap between general-purpose reasoning and specialized real-time control is still vast.