Ammaar-Alam/minebench

MineBench – AI Spatial‑Reasoning Benchmark

What it is – MineBench is a web‑app that lets you evaluate large language models (LLMs) on the ability to reason about three‑dimensional space. A model receives a natural‑language prompt such as “a medieval castle with four towers” and must output raw voxel coordinates (JSON) or call a tiny set of primitives (block, box, line) via the voxel.exec tool. The system then visualises the builds, lets humans compare them head‑to‑head, and produces a ranking using a Bradley‑Terry model with confidence intervals.

Why it matters – Most LLM benchmarks focus on text correctness. MineBench tests a different facet of intelligence: the capacity to construct coherent 3‑D geometry from a description. The authors report that performance on this task correlates with overall model capability, making it a useful “vibe check” for new checkpoints.

Key components

• Arena – Blind, pairwise comparisons of pre‑generated builds; results are aggregated into a leaderboard.
• Sandbox – Upload or generate builds, import outputs from any model, and experiment locally.
• Gallery – Community‑curated prompts and saved generations, searchable and shareable.
• Leaderboard – Live win/loss/draw statistics for every supported model.
• Exports – Download builds in GLB, STL, MagicaVoxel .vox, or WorldEdit .schem formats.

Supported models – OpenAI, Anthropic, Google, Moonshot, DeepSeek, MiniMax, xAI, Z.AI, Qwen, Meta, and any model reachable through OpenRouter.

Getting started locally

# Prerequisites
#   Node.js 18+, pnpm, Docker
pnpm install
cp .env.example .env
pnpm dev:setup   # builds Docker containers, seeds data

In another terminal run the import helper:

pnpm prompt --import

Then open a browser:

• http://localhost:3000/ – Arena
• http://localhost:3000/sandbox – Sandbox
• http://localhost:3000/leaderboard – Leaderboard

Full setup, deployment, and API details are in the docs/ folder.

How models generate builds – Models either return a JSON list of coordinates or, in tool mode, invoke voxel.exec with the minimal primitives. This avoids token‑length limits for large constructions.

Ranking methodology – Because there is no single correct answer, MineBench uses blind human votes between two builds. Votes feed a global Bradley‑Terry model, yielding win probabilities and uncertainty intervals for each model.

Community & contribution – Users can submit new prompts, add model adapters, improve the UI, or fix bugs via pull requests. See CONTRIBUTING.md for guidelines.

Live demo – Try it at https://minebench.ai (and an alpha version at https://alpha.minebench.ai).

---

Bottom line – MineBench provides a concrete, visual way to compare LLMs on spatial reasoning, complementing traditional text‑only benchmarks and offering a playground for researchers and developers interested in 3‑D generation capabilities.