NVlabs/SpatialClaw

📚 What is SpatialClaw?

SpatialClaw is a training‑free framework for spatial‑reasoning agents. It lets a vision‑language model (VLM) write and run Python code step‑by‑step inside a persistent Jupyter kernel that already contains perception tools (e.g., SAM‑3 segmentation, Depth‑Anything‑3 3‑D reconstruction) and scientific libraries (NumPy, SciPy, Matplotlib). The agent can inspect intermediate results, compose tool outputs, and revise its plan before finally returning an answer with ReturnAnswer(...).

The authors evaluate the system on 20 different spatial‑reasoning benchmarks (static images, multi‑view scenes, video/4‑D tasks) and report an average accuracy of 59.9 %, which is +11.2 points over the previous best spatial agent, using the same prompt, tool set, and hyper‑parameters across all benchmarks and six VLM back‑bones (from 26 B to 397 B parameters).

🔧 How it works

Three‑service runtime – a vLLM server (the VLM), a GPU‑accelerated perception‑tool server (segmentation, depth, geometry), and the agent itself (a Jupyter kernel). They communicate via JSON registries and can be run on a single GPU machine or on a SLURM cluster.
Five‑stage loop per sample
- Planner proposes a high‑level strategy.
- The VLM writes one Python cell that may call perception tools, create variables, plot, etc.
- The cell is AST‑checked for safety and executed in the persistent kernel.
- The kernel’s stdout, new variables, and any show() images are sent back as the next observation.
- The loop repeats until the VLM emits ReturnAnswer(...).
Stateful kernel – all variables and loaded images stay alive across steps, so later cells can reuse earlier results without re‑computing.
Tool wrappers – thin Python wrappers expose SAM‑3, Depth‑Anything‑3, and geometry utilities as simple function calls.
LangGraph workflow – the overall orchestration is built on LangGraph, making the loop easy to extend.

🚀 Quick start (single‑machine, no SLURM)

# 1. Clone (including submodules) and install environments (≈15‑30 min)
git clone --recursive https://github.com/NVlabs/SpatialClaw.git
cd SpatialClaw
bash spatial_agent/scripts/setup.sh

# 2. Provide API keys (or run a self‑hosted vLLM that needs none)
cp .env.example .env   # edit the file with your keys

# 3. Run a single benchmark example
python -m spatial_agent.entrypoints.run \
    --dataset spatial_agent/config/dataset/erqa.json \
    --model   spatial_agent/config/model/gemini-3-pro.json \
    --concurrency 4

For cluster runs, see docs/installation.md and docs/running.md for SLURM launch managers and weight‑pre‑download steps.

📂 Repository layout (high‑level)

spatial_agent/ – core agent code, LangGraph workflow, Jupyter‑kernel manager, AST safety checker.
tools/ – wrappers for perception modules (SAM‑3, Depth‑Anything‑3) and geometry utilities.
docs/ – detailed guides (installation, running experiments, monitoring, configuration, architecture).
spatial_agent/config/ – JSON configs for each of the 20 benchmarks and for model back‑ends.
scripts/ – helper scripts for environment setup and SLURM job submission.

🎯 When would you use SpatialClaw?

Research on spatial reasoning – if you need a baseline that can flexibly combine vision tools without fine‑tuning a model.
Tool‑augmented agents – the code‑as‑action interface offers more expressive power than fixed tool‑call APIs.
Benchmarking – the repo ships loaders for 20 standard spatial benchmarks, making reproducibility straightforward.
Rapid prototyping – you can replace or add perception tools by writing new Python wrappers; the agent will automatically be able to call them.

⚠️ Limitations & considerations

Training‑free only – performance depends entirely on the underlying VLM; no fine‑tuning is performed.
GPU‑heavy – the perception‑tool server and the VLM both require GPU memory; running many concurrent samples may need a multi‑GPU setup or SLURM.
Safety checks are static – the AST validator catches obvious unsafe code, but the kernel still executes arbitrary Python, so it should be run in a trusted environment.
License – NVIDIA Source Code License‑NC (non‑commercial). Third‑party tools have their own licenses.

📖 Citation

If you use SpatialClaw in research, please cite:

@article{cho2026spatialclaw,
  title   = {SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning},
  author  = {Cho, Seokju and Hachiuma, Ryo and Badki, Abhishek and
             Su, Hang and Lee, Byung-Kwan and Song, Chan Hee and
             Liu, Sifei and Radhakrishnan, Subhashree and Kim, Seungryong and
             Wang, Yu-Chiang Frank and Chen, Min-Hung},
  journal = {arXiv preprint},
  year    = {2026}
}

TL;DR – SpatialClaw is a research‑grade, code‑driven agent framework that lets large vision‑language models perform sophisticated 2‑D/3‑D/4‑D reasoning by writing Python cells that call powerful perception tools, achieving state‑of‑the‑art results on a wide suite of spatial benchmarks without any model training.