haoz19/world-tracing

Official inference code for World Tracing (NeurIPS 2026 Spotlight). Project page: https://haoz19.github.io/world-tracing-page/

What it solves

World Tracing addresses the limitation of standard mono-depth maps, which only capture the visible surface of an object or scene. It predicts "layered geometry," allowing the system to estimate the 3D structure of both visible and partially occluded surfaces behind the foreground.

How it works

The project uses a flow-matching diffusion model to predict multiple registered XYZ maps (point clouds) from a single image. A single forward pass produces several layers of geometry, creating a 3D scaffold that extends beyond the visible surface. It supports three primary modes:

  • Object mode: Predicts geometry for a single object from an RGBA image.
  • Scene mode: Predicts geometry for entire environments from RGB images.
  • Dynamic mode: Predicts 3D point clouds over time for 16-frame video clips.

Who it’s for

This tool is designed for researchers and developers working in 3D vision, generative AI, and image-to-3D reconstruction. It is particularly useful for those needing a richer 3D representation than a simple depth map for downstream tasks like mesh generation.

Highlights

  • Beyond-Visible Geometry: Predicts occluded surfaces to provide a more complete 3D structure.
  • Multilayer Output: Generates multiple XYZ maps that together form a comprehensive 3D scaffold.
  • Video Support: Capable of predicting dynamic 3D geometry for short video clips.
  • Mesh Integration: Includes a bridge to TRELLIS.2 to convert predicted point clouds into textured GLB meshes.
  • Intrinsic Recovery: Can solve for camera intrinsics directly from the predicted geometry without needing external pose estimators.

Related

  • Project
    lukasHoel/video_to_world

    A 3D reconstruction pipeline that resolves geometric inconsistencies in videos generated by video diffusion models to create stable 3D worlds.

  • Project
    NVlabs/GraspGen

    A diffusion-based framework for 6-DOF robotic grasp generation that supports multiple gripper types and provides real-time, collision-aware grasp predictions from 3D point clouds or meshes.

  • Project
    inspatio/querysplat

    QuerySplat is a 3D Gaussian Splatting prediction framework that decouples geometry and appearance representations to improve the quality of 3D scene reconstructions from images.

  • Project
    guochengqian/Magic123

    A system for generating high-quality 3D objects from a single 2D image by combining 2D and 3D diffusion priors in a coarse-to-fine pipeline.

  • Project
    caiyuanhao1998/Open-DiffusionGS

    A single-stage image-to-3D generation and reconstruction framework that integrates Gaussian Splatting into a diffusion denoiser for fast, scalable 3D creation.