GPT-6 Astra Robot Arm Benchmark: 95% Success on Block Placement vs. 40% for Claude Fable 5.1

Bottom‑line performance

GPT‑6 Astra completed the "block‑into‑bowl" task in 19/20 trials (95% success) while using only $0.94 per run and 2.5 min per trial, outperforming Claude Fable 5.1 (8/20, $2.12, 6.8 min) and Claude Fable 5 (1/20, $2.69, 8.2 min). On the more complex "puzzle‑into‑groove" task, Astra achieved the same 2/20 success rate as Fable 5.1, stalling at the final insertion step.

Why the bowl task matters

The bowl task isolates basic pick‑and‑place competence: grasp, lift, transport, and release.
Astra’s mean stage score of 3.95 (out of 4) shows it consistently reached the final placement stage, whereas Fable 5.1 averaged 2.40 and Fable 5 only 1.30.
Token usage dropped dramatically: Astra used ~2 k output tokens per run versus ~13 k for Fable 5, translating to lower inference cost.

Puzzle‑in‑groove task reveals limits

The puzzle requires precise alignment of a round piece into a circular groove, testing fine‑grained spatial reasoning.
Both Astra and Fable 5.1 reached a mean stage of 2.00 and 2.35, respectively, indicating they could approach the groove but failed to complete insertion.
Astra’s cost per run ($1.36) remained lower than Fable 5.1 ($2.18), but success rates were identical (10%).

Experimental setup

Hardware: Bimanual I2RT YAM arms (6‑DoF each) with parallel‑jaw grippers, observed via three cameras (top, left wrist, right wrist) plus proprioception.
Control: Absolute end‑effector poses (move_to) fed to the robot’s IK solver.
Policy: agent policy from the Inspect‑Robots harness (v0.58.0) with medium reasoning effort, a 20‑LLM‑call budget, and a 25 % speed cap.
Trials: 20 runs per model per task; bowl trials for Astra on rig‑1, for Fable models on rig‑3; puzzle trials on rig‑4 for all models.
Scoring: Human grader assigned a stage (0–4) per trial; the rubric matches the prior Fable report.

Cost and latency breakdown

Model	Task	Avg. output tokens	Est. $/run	Avg. time (min)
GPT‑6 Astra	Bowl	2.1 k	$0.94	2.5
Claude Fable 5.1	Bowl	12.9 k	$2.12	6.8
Claude Fable 5	Bowl	19.2 k	$2.69	8.2
GPT‑6 Astra	Puzzle	2.7 k	$1.36	3.4
Claude Fable 5.1	Puzzle	10.5 k	$2.18	5.9
Claude Fable 5	Puzzle	16.3 k	$2.63	7.9

Costs are calculated at list price ($10 / M input tokens, $50 / M output tokens) and do not account for OpenAI’s automatic input caching, which would further reduce Astra’s effective cost.

Community insights from Hacker News

@baron816 – Suggests focusing on public‑good robotics (e.g., sidewalk trash collection) as an early market to demonstrate value and build trust.

@gizmodo59 – Praises Astra’s speed and versatility for computer‑use tasks, noting a $200 subscription feels worthwhile.

@scronkfinkle – Questions when LLMs will handle everyday chores like laundry, highlighting a gap between impressive demos and practical household automation.

@yurimo – Criticizes the limited scope of the experiment, pointing out the reliance on an external IK controller and the lack of comparison to end‑to‑end VLA/WAM architectures.

@SillyUsername – Reports a negative experience with Astra on a small open‑source robot arm, citing higher cost and poor code generation.

@sheeshkebab – Summarizes the result as “semi‑consistent block moving at the level of a 1‑year‑old child,” underscoring the modest but real progress.

Limitations to keep in mind

Temporal separation – Astra’s runs occurred two days after the Fable runs; environmental changes could affect performance.
Rig mismatch – Bowl trials for Astra used a different physical rig than the Fable trials, potentially influencing success rates.
Human grading bias – Graders knew which model produced each run, opening the possibility of unconscious bias.
Cost inflation – List‑price token costs were used; OpenAI’s caching likely makes Astra’s true cost lower, while Anthropic runs did not benefit from caching.
Manual reset – Objects were repositioned by hand between trials, introducing variability.
Reasoning effort – All models operated at medium effort; higher effort settings might change outcomes.

What this tells us about LLM‑driven robotics

Efficiency gains – Astra’s token‑efficient prompting and faster inference translate into markedly lower per‑run costs for simple manipulation.
Task specificity – Success spikes on straightforward pick‑and‑place but plateaus on tasks requiring fine alignment, indicating current LLM planners lack the precision needed for intricate assembly.
Scalability concerns – Even a $1‑$2 cost per simple pick‑and‑place is still high for large‑scale deployment; hardware advances or model optimization will be needed to reach economically viable rates.
Modular pipelines – The experiment underscores the value of separating high‑level planning (LLM) from low‑level control (IK), but also hints at future gains from tighter integration (e.g., VLM‑to‑VLA architectures).

Bottom line for practitioners

Deploy GPT‑6 Astra for low‑complexity pick‑and‑place where speed and cost matter, but do not expect it to handle fine‑grained insertion without additional control layers.
Consider public‑service pilots (e.g., trash collection) to showcase capability while gathering real‑world data.
Anticipate cost reductions as model pricing improves and caching becomes more transparent.
Keep an eye on modular robotics stacks that combine LLM planners with specialized low‑level controllers for tasks beyond simple grasping.