NVlabs/PointWorld

PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation

What it solves

PointWorld addresses the challenge of scaling 3D world models for robotic manipulation in real-world ("in-the-wild") environments. It provides a way to predict how a full 3D scene will evolve (scene flow) based on partial RGB-D captures and the actions the robot takes.

How it works

PointWorld is a large pre-trained 3D world model that represents both the environment and robot actions as 3D point flows. It predicts full-scene 3D point flows from partially observable RGB-D data. The architecture utilizes a scene encoder backbone based on DINOv3 and incorporates adapted components from Point Transformer V3 (PTv3) in various sizes (small, base, and large).

Who it’s for

This project is designed for researchers and developers working on 3D world models, robotic manipulation, and embodied AI who need a pipeline for training and evaluating 3D scene-flow predictions.

Highlights

  • Multi-Domain Training: Supports training on DROID (real-world) and BEHAVIOR (simulation) datasets, or a combination of both.
  • 3D Point Flow Representation: Predicts full-scene evolution by treating both observations and actions as 3D point flows.
  • Interactive Visualization: Includes a live 3D viewer built on viser to step through temporal evolution and compare model predictions against ground truth.
  • Flexible Architecture: Offers multiple PTv3 size variants (small, base, large) to balance performance and resource usage.

Related

  • Project
    yuxumin/PoinTr

    PoinTr is a transformer-based model for point cloud completion that reconstructs full 3D objects from partial point clouds using a geometry-aware encoder-decoder architecture.

  • Project
    Pointcept/Utonia

    A cross-domain pre-trained Point Transformer V3 encoder for 3D point clouds, providing a universal representation for various downstream 3D perception tasks.

  • Project
    Pointcept/Concerto

    A joint 2D-3D self-supervised pre-trained Point Transformer V3 model for extracting high-quality spatial representations from 3D point clouds.

  • Project
    Visionary-Laboratory/holi-spatial

    A data curation pipeline that transforms video streams into annotated 3D spatial intelligence datasets, including 3D Gaussian Splatting geometry and spatial QA pairs.

  • Project
    AlayaLab/WildWorld

    A large-scale action-conditioned world modeling dataset with explicit state annotations from a AAA ARPG, designed for training generative game environments.