Visionary-Laboratory/holi-spatial

What it solves

Holi-Spatial is designed to transform raw video streams into a comprehensive 3D spatial intelligence system. It addresses the challenge of creating high-quality, annotated 3D scenes from video, enabling models to understand spatial relationships, object instances, and functional regions within a 3D environment.

How it works

The project implements a multi-stage pipeline to process video data from datasets like ScanNet, ScanNet++, and DL3DV:

1. Geometric Optimization: Uses Depth Anything 3 (DA3) for depth and point cloud initialization, followed by PGSR/3DGS training to create 3D Gaussian Splatting geometry.
2. Image-level Perception: Employs VLMs (Vision-Language Models) for discovering object classes and regions, and SAM3 for generating precise image masks.
3. Scene-level Lift and Refinement: Lifts 2D masks into 3D space, merges and filters 3D bounding boxes, generates captions for instances, and synthesizes spatial Question-and-Answer (QA) pairs.

Who it’s for

Researchers and developers working on 3D vision, embodied AI, and multimodal LLMs who need a pipeline to generate richly annotated 3D spatial datasets for training vision-language models.

Highlights

• End-to-End Pipeline: Converts video streams into 3DGS geometry, meshes, and 3D bounding box annotations.
• Large-scale Dataset Generation: Capable of producing millions of spatial QA pairs (e.g., the HoliSpatial-QA-2M dataset).
• Integration with Foundation Models: Leverages SAM3, Depth Anything 3, and Qwen3-VL for perception and annotation.
• SFT Ready: Includes tools to convert generated QA data into formats compatible with LLaMA-Factory for supervised fine-tuning.