AIGeeksGroup/3D-R1

3D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understanding

What it solves

3D-R1 addresses the limitations of current 3D Vision-Language Models (VLMs), which often struggle with robust reasoning and generalization due to a lack of high-quality spatial data and static viewpoint assumptions. It aims to provide a unified approach to 3D scene understanding, including tasks like dense captioning, visual grounding, and 3D question answering.

How it works

3D-R1 employs a multi-stage training and architectural approach:

  • Cold-Start Initialization: It uses Scene-30K, a high-quality synthetic dataset featuring Chain-of-Thought (CoT) reasoning, generated via a data engine based on Gemini 2.5 Pro.
  • Reinforcement Learning (RL): The model is further refined using RLHF policies like GRPO. It utilizes three specific reward functions—perception, semantic similarity, and format rewards—to ensure detection accuracy and semantic precision.
  • Dynamic View Selection: Instead of relying on static viewpoints, it adaptively selects the most informative perspectives for understanding a 3D scene.
  • Backbone: It leverages the Qwen2.5-VL-7B-Instruct model as a base LLM.

Who it’s for

This project is designed for researchers and developers working on 3D computer vision, spatial AI, and the integration of large language models with 3D scene perception.

Highlights

  • Unified Scene Understanding: Capable of handling 3D-DC (Dense Captioning), 3D-VG (Visual Grounding), 3D-QA (Question Answering), 3D Dialogue, 3D Reasoning, and 3D Planning.
  • Synthetic CoT Data: Introduces Scene-30K to bridge the gap in high-quality 3D spatial reasoning data.
  • RL-Enhanced Reasoning: Uses GRPO and specialized reward functions to improve reasoning capabilities.
  • Zero-Shot Generalization: Demonstrates strong performance on complex scenes without task-specific training.

Related

  • Project
    om-ai-lab/VLM-R1

    VLM-R1 is a framework for training R1-style Large Vision-Language Models using GRPO to improve reasoning and generalization in tasks like visual grounding, math, and object detection.

  • Project
    Wakals/CoVT

    CoVT is a framework that enables Vision-Language Models to reason using continuous visual tokens, improving their spatial reasoning and geometric awareness by grounding semantic thoughts in perceptual cues.

  • Project
    facebookresearch/VLM3

    VLM³ is a framework that enables standard Vision Language Models to perform 3D vision tasks like depth estimation and camera pose estimation using only text-based outputs and data scaling.

  • Project
    microsoft/World-R1

    World-R1 is a reinforcement learning framework that improves 3D geometric consistency in text-to-video generation by aligning generated videos with 3D constraints without changing the base model architecture.

  • Project
    SkyworkAI/Skywork-R1V

    Skywork-R1V is an open-source multimodal reasoning model that uses reinforcement learning and visual chain-of-thought to achieve state-of-the-art performance in complex visual logic, math, and physics tasks.