WHU-USI3DV/VistaDream

VistaDream – Reconstructing 3D Scenes from a Single Photo

What it does
VistaDream is a research‑grade Python library (PyTorch) that can turn a single RGB image into a full 3‑D scene. It does this without any extra training – it re‑uses existing diffusion‑based image generators and depth estimators, then adds a training‑free multi‑view consistency step so that the synthesized novel views line up with each other.

Why it matters
Most current 3‑D reconstruction pipelines need many calibrated photos or a costly optimisation loop. VistaDream shows that, by cleverly sampling diffusion models, you can get high‑quality, multiview‑consistent renderings from just one picture, opening the door to quick content creation for AR/VR, games, and visual effects.

Key components

Stage	What happens	Main tools used
1️⃣ Global scaffold	A coarse 3‑D layout is built by out‑painting the image borders and estimating a depth map.	Fooocus (diffusion inpainting), Depth‑Pro / Prior‑Depth‑Anything (depth), optional VGGT for sparse‑view scaffolds
2️⃣ Iterative RGB‑D inpainting	The scaffold is filled with novel‑view RGB‑D images using diffusion‑based inpainting.	Same diffusion models as above
3️⃣ Multi‑view Consistency Sampling (MCS)	During the reverse diffusion step, extra constraints force all generated views to agree, yielding a coherent scene.	Purely algorithmic – no extra training

How to try it

1. Install the environment (the repo suggests using a coding‑assistant like Codex to avoid dependency headaches).
2. Run bash download_weights.sh to fetch the required pretrained checkpoints (Fooocus, Depth‑Pro, etc.).
3. Execute python demo.py for a single‑view demo or python demo_sparse.py if you have a few sparse images.
4. The output includes a gaussian‑field file (scene.pth) and an MP4 video of RGB (and depth) renderings. The field can be exported to a PLY file and visualised in the SuperSplat web viewer.

What you need

• Ubuntu 20.04 (tested)
• One high‑end GPU (e.g., RTX 4090) because diffusion inference is GPU‑intensive.
• Conda environment vdplus (see INSTALL.md).

Current status

• Accepted at ICCV 2025, journal version in IEEE TPAMI (2026).
• The code is training‑free but relies on several external pretrained models.
• A branch (moge3‑lbd) adds upgraded depth modules (MoGe‑3, LingBot‑Depth) for better quality.

Who might use it

• Researchers exploring diffusion‑based 3‑D synthesis.
• Creators who want to prototype a 3‑D asset from a single reference photo.
• Developers building AR/VR pipelines that need fast, on‑the‑fly scene generation.

---

All information above is taken directly from the repository’s README; no additional features have been assumed.