BAAI-DCAI/SpatialBot

The official repo for "SpatialBot: Precise Spatial Understanding with Vision Language Models.

What it solves

SpatialBot is designed to improve the precise spatial understanding of Vision Language Models (VLMs). It addresses the gap in how AI models perceive depth and spatial relationships in images, enabling them to handle complex spatial reasoning tasks and robot manipulation.

How it works

SpatialBot is a multi-image VLM based on the Bunny architecture. It can process multiple input images, such as combining an RGB image with a depth map to better understand the 3D structure of a scene. It is trained on the SpatialQA dataset and can be further fine-tuned using LoRA (Low-Rank Adaptation) on specific datasets like SpatialQA-E for robot manipulation tasks.

Who it’s for

This project is intended for researchers and developers working in robotics, computer vision, and embodied AI, specifically those needing models that can accurately reason about spatial coordinates and depth.

Highlights

  • Multi-image support: Capable of processing 0-8 images (RGB and depth maps) to provide spatial context.
  • RGB-D integration: Specifically optimized for RGB-Depth inputs to enhance spatial precision.
  • Embodiment capabilities: Can be used as a Vision-Language-Action (VLA) model for pick-and-place robot manipulation by predicting delta positions or key points.
  • Comprehensive ecosystem: Includes the SpatialQA training set, the SpatialBench benchmark, and the SpatialQA-E embodiment dataset.

Related

  • Project
    remyxai/VQASynth

    A pipeline for transforming image datasets into spatial VQA datasets to improve the 3D spatial reasoning and distance estimation capabilities of Vision-Language Models.

  • Project
    dexmal/opendm

    OpenDM is a Vision-Language-Action (VLA) model framework for open-world robot control, providing tools for training, fine-tuning, and deploying models that can handle complex, multi-embodiment tasks.

  • Project
    Wakals/CoVT

    CoVT is a framework that enables Vision-Language Models to reason using continuous visual tokens, improving their spatial reasoning and geometric awareness by grounding semantic thoughts in perceptual cues.

  • Project
    cambrian-mllm/cambrian-s

    A suite of multimodal LLMs and datasets designed to enhance spatial reasoning and "spatial supersensing" in video understanding.

  • Project
    Robbyant/lingbot-vla-v2

    LingBot-VLA 2.0 is a Vision-Language-Action foundation model that enables robots to generalize across different embodiments and tasks using a unified action representation and MoE action experts.