InternRobotics/PointLLM

[ECCV 2024 Best Paper Candidate & TPAMI 2025] PointLLM: Empowering Large Language Models to Understand Point Clouds

What it solves

PointLLM addresses the challenge of enabling Large Language Models (LLMs) to understand 3D colored point clouds. It allows models to perceive object types, geometric structures, and appearance without being hindered by issues like ambiguous depth, occlusion, or viewpoint dependency.

How it works

PointLLM uses a point encoder to extract features from input point clouds and projects them into the latent space of an LLM backbone. The LLM then processes these point tokens alongside text tokens to generate descriptive or classificatory text outputs. The model is trained using a two-stage strategy on a novel dataset of 660K simple and 70K complex point-text instruction pairs.

Who it’s for

This project is designed for researchers and developers working in 3D computer vision, robotics, and multi-modal AI who need to integrate 3D spatial understanding with the reasoning capabilities of LLMs.

Highlights

  • Multi-modal 3D Understanding: Capable of understanding colored point clouds for object classification and captioning.
  • Large-scale Instruction Dataset: Includes a dataset of 730K point-text instruction pairs to facilitate training.
  • Comprehensive Benchmarks: Establishes benchmarks for Generative 3D Object Classification and 3D Object Captioning.
  • Flexible Evaluation: Supports evaluation via traditional metrics and LLM-based evaluation (ChatGPT/GPT-4).
  • Ready-to-use Checkpoints: Provides pre-trained weights for 7B and 13B parameter versions.

Related

  • Project
    PointCloudLibrary/pcl

    A large-scale open-source library for 2D/3D image and point cloud processing, used widely in robotics and 3D perception.

  • Project
    ch-sa/labelCloud

    A lightweight 3D bounding box labeling tool for point clouds, used to generate training data for 3D object detection and 6D pose estimation.

  • Project
    facebookresearch/VLM3

    VLM³ is a framework that enables standard Vision Language Models to perform 3D vision tasks like depth estimation and camera pose estimation using only text-based outputs and data scaling.

  • Project
    cambrian-mllm/cambrian-s

    A suite of multimodal LLMs and datasets designed to enhance spatial reasoning and "spatial supersensing" in video understanding.

  • Dispatch
    Vision Language Models Explained

    Hugging Face provides a comprehensive guide to Vision Language Models (VLMs), detailing their architecture, open-source options, evaluation benchmarks, and new experimental support for fine-tuning via the TRL library.