lukasHoel/video_to_world

Our method reconstructs 3D worlds from video diffusion models using non-rigid alignment to resolve inherent 3D inconsistencies in the generated sequences.

What it solves

This project addresses the problem of 3D inconsistencies in videos generated by video diffusion models. While these models can create visually impressive sequences, they often lack geometric consistency, making it difficult to reconstruct a stable 3D world from them. This tool resolves those inconsistencies using non-rigid alignment to create a coherent 3D reconstruction.

How it works

The system follows a multi-stage pipeline to transform a video into a 3D scene:

  1. Preprocessing: Uses DepthAnything-3 to estimate per-frame point clouds from the input video.
  2. Non-rigid Alignment: Employs an iterative Frame-to-model ICP (Iterative Closest Point) process to align these inconsistent per-frame point clouds into a single canonical frame, creating deformation fields.
  3. Global Optimization: Jointly refines all per-frame deformations to sharpen the resulting canonical point cloud.
  4. Inverse Deformation Training: Trains a network to map points from the canonical space back into each frame's camera space.
  5. Gaussian Splatting: Optimizes a 2D or 3D Gaussian Splatting (GS) scene based on the canonical point cloud, using the inverse deformation network to handle frame-specific warping during training.

Who it’s for

Researchers and developers working with AI-generated video, 3D reconstruction, and computer vision who want to turn inconsistent video diffusion outputs into stable 3D environments.

Highlights

  • Handles Inconsistency: Specifically designed to fix the geometric "drift" and inconsistencies inherent in video diffusion models.
  • Flexible Rendering: Supports both 2D and 3D Gaussian Splatting for the final scene representation.
  • End-to-End Pipeline: Provides a full workflow from raw MP4 input to a final 3D PLY export or interactive visualization.
  • Integration: Leverages state-of-the-art tools like DepthAnything-3 for depth estimation and gsplat for rendering.

Related

  • Project
    scu-zjz/IMDLBenCo

    A comprehensive benchmark and modular codebase for image manipulation detection and localization, providing standardized components and SOTA model implementations.

  • Project
    PRBonn/RAP

    A 3D point cloud registration framework that uses flow matching to align multiple unposed scans into a common frame by treating registration as a conditional generation task.

  • Project
    nianticlabs/map-free-reloc

    A reference implementation for map-free visual relocalization that enables metric pose estimation relative to a single image, removing the need for large 3D maps.

  • Project
    fabiotosi92/ZipDepth

    ZipDepth is a 6 M‑parameter, zero‑shot monocular depth model that runs at >1 k FPS on GPUs and can be exported to mobile runtimes. It combines RepVGG‑style re‑parameterizable convolutions, strip‑pooling attention, and a lightweight FPN decoder. The repo provides ready‑to‑run inference scripts, ONNX/TorchScript export utilities, benchmarks, and full training pipelines (knowledge‑distilled from Depth Anything V2).