WHU-USI3DV/VistaDream

What it solves

VistaDream addresses the challenge of reconstructing high-quality 3D scenes from a single-view image. While existing diffusion models can generate novel views, they often lack consistency between those generated images, leading to fragmented or unrealistic 3D reconstructions.

How it works

VistaDream uses a training-free, two-stage pipeline to ensure multiview consistency:

1. Global Scaffold Construction: It creates a coarse 3D scaffold by combining an estimated depth map with outpainted boundaries (zooming out slightly from the original image).
2. Iterative RGB-D Inpainting: It uses diffusion-based inpainting to fill the holes in the scaffold with novel-view images.
3. Multi-view Consistency Sampling (MCS): To further refine the result, it introduces consistency constraints during the reverse sampling process of the diffusion model, ensuring that the generated views align logically without requiring additional training or fine-tuning.

Who it’s for

This project is for researchers and developers working in 3D scene reconstruction, novel-view synthesis, and generative AI, specifically those looking for a way to turn a single 2D image into a consistent 3D environment.

Highlights

• Training-Free: Works with existing pretrained diffusion models without needing further training or fine-tuning.
• Multiview Consistency: Specifically solves the problem of inconsistency between generated novel views.
• Flexible Input: Supports both single-view and sparse-view image inputs.
• Interactive Interface: Includes an interactive demo for scene generation.