AIGeeksGroup/3D-R1

3D‑R1 – A Generalist 3‑D Vision‑Language Model

What it is – 3D‑R1 is an open‑source foundation model that extends large vision‑language models (VLMs) into the 3‑D domain. It is trained to reason about point‑cloud scenes, answer questions, generate captions, ground objects, and even plan actions, all from a single model.

Why it matters – Existing 3‑D VLMs often falter on complex reasoning because they rely on static viewpoints and limited spatial data. 3D‑R1 tackles this by:

• Pre‑training on a synthetic, chain‑of‑thought (CoT) dataset called Scene‑30K that supplies high‑quality 3‑D scenes with rich language annotations.
• Using reinforcement‑learning‑with‑human‑feedback (RLHF) (GRPO) with three custom rewards (perception, semantic similarity, format) to sharpen reasoning and answer fidelity.
• Dynamically selecting the most informative camera views during inference, which improves generalisation across unseen scenes.

Key components

Component	Role
Scene‑30K	Synthetic 30 K 3‑D scenes with CoT captions; serves as cold‑start data
RLHF (GRPO)	Fine‑tunes the model with perception, semantic‑similarity and format rewards
Dynamic view selector	Chooses optimal viewpoints for each scene
Backbone LLM	Qwen2.5‑VL‑7B‑Instruct (or compatible) provides the language generation side
3‑D encoders	PointNet++ (built from third_party/pointnet2) and optional PyTorch3D rendering
Visualization	Rerun‑SDK based viewer for point clouds and 3‑D bounding‑box predictions

Getting started

1. Environment – Tested with CUDA 11.8, Python 3.9, PyTorch 2.0.1+cu118. Install the listed pip/conda packages (including optional pytorch3d). Build PointNet++ and the accelerated GIoU ops as shown in the README.
2. Data – Download the pre‑processed ScanNet data from the provided Hugging Face link, then follow the instructions to add multi‑view images/depth maps if you want to enable the extra encoders. Pull language annotations from ScanRefer, Nr3D, ScanQA, and 3D‑LLM, or simply download the ready‑made Scene‑30K dataset.
3. Training –
   • Supervised fine‑tuning on Scene‑30K: bash script/train.generalist.sh
   • RL fine‑tuning: bash script/train.rl.sh
4. Inference – Run bash script/infer.sh with a checkpoint to evaluate on the validation split of the benchmark datasets.
5. Visualization – Install rerun-sdk, trimesh, plyfile and run bash script/visualize.sh to see point‑clouds and predicted 3‑D boxes.

Demo & resources

• Paper: https://arxiv.org/abs/2507.23478 (NeurIPS 2026 submission)
• Project page: https://aigeeksgroup.github.io/3D‑R1
• Datasets: Scene‑30K on Hugging Face
• Pre‑trained models: Hugging Face repo AIGeeksGroup/3D‑R1
• YouTube overview: https://youtu.be/P3jPg6MwLOM

Use cases – Zero‑shot 3‑D dense captioning, object captioning, visual grounding, question answering, dialogue, reasoning, and planning on point‑cloud scenes. The README showcases several video demos of these capabilities.

Acknowledgements – Builds on Qwen, LSceneLLM, ARKitScenes, and DeepSeek‑Math codebases.

---

If you reuse the code or cite the work, please add the provided BibTeX entry.