remyxai/VQASynth

Compose multimodal datasets 🎹

What it solves

VQASynth is designed to address the scarcity of high-quality spatial reasoning data in AI pretraining datasets. It allows users to transform standard image datasets into Visual Question Answering (VQA) datasets that specifically focus on 3D spatial relationships, distances, and orientations, which are critical for embodied AI and robotics.

How it works

The project implements a pipeline that fuses semantic and metric data to generate synthetic spatial VQA pairs. The process involves:

  1. Localization: Using tools like Florence-2 and SAM2 to detect and segment task-relevant objects.
  2. 3D Reconstruction: Lifting images to 3D using VGGT to create per-object point clouds, depth maps, and intrinsics.
  3. VQA Generation: Using a prompt generator to create question-answer pairs based on the reconstructed 3D scene, often incorporating Chain-of-Thought (CoT) reasoning to improve accuracy.

Additionally, the project provides an agent-based approach (via NOOA) that can dynamically compose tool calls (e.g., metric depth, object orientation, regional captioning) to answer spatial questions on the fly without following a pre-templated pipeline.

Who it’s for

  • AI Researchers: Those working on Vision-Language Models (VLMs) and spatial reasoning.
  • Robotics Developers: Engineers building embodied AI that needs to understand 3D distances and object orientations in physical environments.
  • Dataset Curators: Users who want to augment existing image datasets with grounded spatial annotations.

Highlights

  • Metric Depth Integration: Replaces DepthPro with VGGT for faster and more accurate metric depth estimation.
  • Synthetic Data Generation: Capable of producing datasets like SpaceOm and SpaceThinker for instruction-tuning VLMs.
  • Dynamic Tool Composition: An agentic version allows for flexible, non-templated spatial reasoning via a wide array of tools (e.g., Orient-Anything, MediaPipe pose keypoints).
  • CoT Reasoning: Generates reasoning traces that allow VLMs to estimate distances based on real-world object priors and visual cues.
  • Lerobot Integration: Compatible with the VlmClient protocol for use in robot-learning pipelines as a reward or CoT source.

Related

  • Project
    Wakals/CoVT

    CoVT is a framework that enables Vision-Language Models to reason using continuous visual tokens, improving their spatial reasoning and geometric awareness by grounding semantic thoughts in perceptual cues.

  • Project
    Visionary-Laboratory/holi-spatial

    A data curation pipeline that transforms video streams into annotated 3D spatial intelligence datasets, including 3D Gaussian Splatting geometry and spatial QA pairs.

  • Project
    AIGeeksGroup/3D-R1

    3D-R1 is a generalist 3D Vision-Language Model that enhances spatial reasoning and scene understanding through synthetic CoT data and reinforcement learning.

  • Project
    gokayfem/ComfyUI_VLM_nodes

    A production-oriented suite of ComfyUI nodes for vision-language models, providing structured detection, segmentation, and adaptive video reasoning with VRAM optimizations.

  • Project
    BAAI-DCAI/SpatialBot

    SpatialBot is a Vision-Language-Action model that enhances spatial understanding and depth perception in VLMs, enabling precise spatial reasoning and robot manipulation.