nv-tlabs/dvlt

📚 Project summary

Déjà View (DVLT) – Looping Transformers for Multi‑View 3D Reconstruction – is a research‑grade library released by NVIDIA together with several universities. It implements a recurrent transformer that can take an unordered set of images or video frames and output:

• Per‑pixel depth maps
• Camera poses (extrinsics & intrinsics)
• 3‑D point clouds (world‑space coordinates)
• Confidence scores for each ray

The model is trained once and can be run for a variable number of refinement steps (K). More steps give higher quality at the cost of extra compute, letting you trade speed vs. accuracy on the fly.

---

🚀 What’s inside the repo?

Component	What it provides
dvlt/model/dvlt	Core recurrent‑transformer implementation and four ablation variants (vanilla, decoupled blocks, no s_out token, no depth‑scaling).
Evaluation wrappers	Thin adapters for five public baselines (VGGT, VGGT‑Omega, Depth‑Anything‑3, MapAnything, Pi3). They import the original packages (installed separately) so you can compare DVLT against state‑of‑the‑art methods.
Training stack	Scripts built on 🤗 accelerate + Hydra, with optional Weights & Biases logging. Includes a Stage‑2 fine‑tune recipe for the depth‑conv head.
Inference & demo	Ready‑to‑run inference code, a Gradio web UI, and an offline mode that writes .glb / .rrd files for downstream visualization.
Data loaders	Pre‑processing pipelines for ScanNet++, DTU, ETH3D, 7Scenes, NuScenes and a lightweight “benchmark_lite” that skips heavy preprocessing.
Docs & tooling	Installation guide, dataset‑setup docs, contribution guide, test suite, and tab‑completion helpers for the CLI scripts.

---

🛠️ Quick start (from the README)

# 1️⃣ Create environment
conda create -n dvlt python=3.12 && conda activate dvlt
conda install pytorch=2.5.1 torchvision pytorch-cuda=12.4 -c pytorch -c nvidia -c conda-forge

# 2️⃣ Install the package (all optional extras)
pip install -e .[all]

Inference example (Python)

import torch
from accelerate import Accelerator
from dvlt.model.dvlt.model import DVLT
from dvlt.util.preprocess import load_sequence, preprocess_images

ckpt = "nvidia/dvlt"          # can be a local folder, URL or HF hub ID
seq_path = "path/to/scene_dir" # or a video file

accelerator = Accelerator(mixed_precision="bf16")
model = DVLT(img_size=504)
model.load_pretrained(ckpt, strict=True)
model.setup_test(accelerator)

_, frames = load_sequence(seq_path)
batch = preprocess_images(frames, img_size=504, patch_size=14,
                           device=accelerator.device)

with torch.no_grad(), accelerator.autocast():
    preds = model.predict(batch, accelerator)

# Access results
cameras = preds["cameras"][0]
extrinsics = cameras.camera_to_worlds          # (S, 3, 4)
intrinsics = cameras.get_intrinsics_matrices()# (S, 3, 3)
depths = preds["depths"][0]                   # (S, H, W)
points = preds["world_points"][0]             # (S, H, W, 3)

Training (single‑GPU)

python -m dvlt.scripts.train --config-name dvlt-large data=scannetpp

Multi‑GPU with accelerate:

accelerate launch --num-processes 4 -m dvlt.scripts.train \
    --config-name dvlt-large data=scannetpp

Resume a run:

python -m dvlt.scripts.train \
    --config-dir=outputs/<run> --config-name=config.yaml \
    trainer.resume_from_checkpoint=latest

Evaluation

# Full benchmark (DTU, ETH3D, 7Scenes, ScanNet++, NuScenes)
python -m dvlt.scripts.test --config-name dvlt data=benchmark

# Lightweight benchmark (no heavy preprocessing)
python -m dvlt.scripts.test --config-name dvlt data=benchmark_lite

Interactive demo (Gradio)

python -m dvlt.scripts.gradio_demo          # opens http://localhost:7860
# Offline mode – writes .glb/.rrd files instead of launching UI
python -m dvlt.scripts.gradio_demo --offline \
    --input /path/to/scene_dir --models dvlt

---

📊 Reported performance (full benchmark)

Dataset	Pose AUC@3	Pose AUC@30	Depth inlier@3%	Depth AbsRel
DTU	0.8319	0.9880	0.9706	0.0093
ETH3D	0.6604	0.9536	0.7717	0.0267
7Scenes	0.1393	0.8172	0.7437	0.0349
ScanNet++	0.7941	0.9803	0.9239	0.0167
NuScenes	0.4340	0.8534	0.5853	0.0673

---

⚙️ Configuration system

The project uses Hydra. Experiment configs live under src/dvlt/config/experiments/.

• dvlt-large – main training recipe (large model, full schedule).
• dvlt-large-ablation* – toggle specific architectural changes via command‑line overrides.
• dvlt-large-depthconv-stage2 – fine‑tune the depth‑conv head (matches the released checkpoint).
• dvlt – inference‑only alias that loads the released Stage‑2 checkpoint.
• Baseline wrappers (vggt, da3‑large, pi3x, …) are also defined here for evaluation.

User‑specific paths (e.g., where datasets are stored) are set in src/dvlt/config/experiments/user/default.yaml. Copy it to local.yaml and edit data_root, then select with user=local on the command line.

📄 License

• Code – Apache 2.0 (see LICENSE).
• Model weights – NVIDIA License (research‑and‑evaluation, non‑commercial only).
• Third‑party components retain their original licenses (listed in THIRD_PARTY_LICENSES.md).

---

📚 How to cite

@article{burzio2026dejaview,
  title   = {D\'ej\`a View: Looping Transformers for Multi-View 3D Reconstruction},
  author  = {Burzio, Alessandro and Fischer, Tobias and Elflein, Sven and Zhou, Qunjie and de Lutio, Riccardo and Ren, Jiawei and Huang, Jiahui and Huang, Shengyu and Pollefeys, Marc and Leal-Taix{\'e}, Laura and Gojcic, Zan and Turki, Haithem},
  journal = {arXiv preprint arXiv:2605.30215},
  year    = {2026}
}

---

🎯 Who might use this?

• Researchers exploring neural 3‑D reconstruction, especially those interested in recurrent or looping transformer architectures.
• Engineers building AR/VR pipelines that need fast, iterative depth and pose estimation from casual photo/video captures.
• Anyone wanting a fair comparison against recent baselines (VGGT, Depth‑Anything‑3, MapAnything, Pi3) in a single codebase.

---

🔗 Links

• Project page: https://research.nvidia.com/labs/dvl/projects/dvlt/
• arXiv paper: https://arxiv.org/abs/2605.30215
• Model on Hugging Face: https://huggingface.co/nvidia/dvlt