Pointcept/Concerto

[NeurIPS'25] Official repository of Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations

What it solves

Concerto provides a way to learn high-quality spatial representations for 3D point clouds using joint 2D-3D self-supervised pre-training. It addresses the challenge of creating robust 3D encoders that can be effectively used for various downstream 3D perception tasks without requiring massive amounts of labeled 3D data.

How it works

Concerto is based on the Point Transformer V3 (PTv3) architecture. It uses a self-supervised learning approach, modified from the Sonata framework, to pre-train the model on both 2D and 3D data. The resulting encoder-only model can then be used to extract features from 3D point clouds (including coordinates, colors, and normals) and map those features back to the original point cloud scale for downstream use.

Who it’s for

This project is designed for researchers and developers working on 3D computer vision, point cloud analysis, and spatial representation learning.

Highlights

  • Joint 2D-3D Pre-training: Leverages both modalities to emerge better spatial representations.
  • Multiple Model Sizes: Offers pre-trained weights for small (39M), base (108M), and large (208M) versions of the model.
  • Flexible Input: Supports point clouds with or without color and normal information.
  • Integrated Visualization: Includes demos for PCA visualization, similarity heatmaps, and semantic segmentation linear probing.

Related

  • Project
    Pointcept/Utonia

    A cross-domain pre-trained Point Transformer V3 encoder for 3D point clouds, providing a universal representation for various downstream 3D perception tasks.

  • Project
    Pointcept/Pointcept

    A powerful and flexible codebase for point cloud perception research, integrating various 3D backbones and self-supervised pre-training frameworks.

  • Project
    NVlabs/PointWorld

    PointWorld is a large pre-trained 3D world model that predicts full-scene 3D point flows from partial RGB-D captures and robot actions to improve in-the-wild robotic manipulation.

  • Project
    yuxumin/PoinTr

    PoinTr is a transformer-based model for point cloud completion that reconstructs full 3D objects from partial point clouds using a geometry-aware encoder-decoder architecture.

  • Project
    InternRobotics/PointLLM

    PointLLM is a multi-modal large language model that enables LLMs to understand colored 3D point clouds for tasks like object classification and captioning.