facebookresearch/vggt-omega

What is VGGT‑Ω?

VGGT‑Ω (pronounced V‑G‑G‑T Omega) is a research‑grade neural network that takes one or more RGB images and simultaneously predicts:

Camera pose (extrinsic + intrinsic parameters) for each view,
Depth maps and a confidence estimate for those depths,
Optional text‑alignment embeddings that tie the visual representation to natural‑language prompts.

The model is built by the Visual Geometry Group at Oxford together with Meta AI and is described in the arXiv paper VGGT‑Ω (arXiv:2605.15195). It is intended for computer‑vision tasks that need geometry – e.g., 3‑D reconstruction, AR/VR, robotics, or any application where you want to know where the camera was and how far away surfaces are, directly from raw photos.

Key Features (as documented in the repo)

Feature	Detail
Model sizes	One 1‑billion‑parameter checkpoint at 512 px resolution (general use) and a 256 px version that includes text‑alignment embeddings.
Training data	Trained on eight large‑scale datasets; the repo now ships the sequence lists and a curated pipeline for preparing new data, including a supervised geometric‑filtering step.
Reproducibility	A second “reproduction” checkpoint (416 px) and accompanying `reproduction.md` are provided to verify benchmark numbers.
Inference output	Returns a dictionary with `pose_enc` (camera encoding), `depth`, `depth_conf`, and `camera_and_register_tokens` (separate camera token and per‑pixel register tokens).
Demo	A Gradio web UI that visualises the predicted point cloud (depth‑unprojected) and camera poses as a GLB scene.
GPU memory profile	Detailed table showing peak memory for up to 500 frames on an A100 (e.g., ~6 GB for a single 512‑px frame).
License	Open‑source under the repository’s LICENSE (see file).

Quick‑Start Guide (from the README)

# 1. Clone and install
git clone git@github.com:facebookresearch/vggt-omega.git
cd vggt-omega
pip install -r requirements.txt
pip install -e .

import torch
from vggt_omega.models import VGGTOmega
from vggt_omega.utils.load_fn import load_and_preprocess_images
from vggt_omega.utils.pose_enc import encoding_to_camera

# Load a checkpoint (you must request access on Hugging Face first)
ckpt = "path/to/vggt_omega_1b_512.pt"
model = VGGTOmega().to("cuda").eval()
model.load_state_dict(torch.load(ckpt, map_location="cpu"))

# Prepare images (list of file paths)
imgs = load_and_preprocess_images(["imgA.png", "imgB.png"], image_resolution=512).to("cuda")

with torch.inference_mode():
    out = model(imgs)

# Convert the pose encoding to usable extrinsics / intrinsics
extrinsics, intrinsics = encoding_to_camera(
    out["pose_enc"], out["images"].shape[-2:]
)

# Depth and confidence maps
depth = out["depth"]
conf  = out["depth_conf"]

For the text‑aligned checkpoint, instantiate the model with VGGTOmega(enable_alignment=True) and read out["text_alignment_embedding"].

Running the Interactive Demo

pip install -r requirements_demo.txt   # extra deps for Gradio & visualisation
python demo_gradio.py \
  --checkpoint checkpoints/VGGT-Omega-1B-512/model.pt \
  --image-resolution 512

The demo lets you upload images or a video, runs the model, and shows the resulting point cloud and camera poses in a 3‑D viewer.

Where to Get the Model

Request access on the Hugging Face model page: https://huggingface.co/facebook/VGGT-Omega. After approval you can download the .pt files listed in the table (512 px, 416 px reproduction, and 256 px text‑aligned).

Reproducibility Resources

training/ – code used for the original training runs.
training/valid_seqs/ – sequence lists for the eight datasets.
training/dataset_preparation/ – scripts and guides for collecting, converting, cleaning, and visually reviewing data (including an agent‑assisted review tool).
training/curation/ – a supervised geometric‑filtering pipeline.
reproduction.md – instructions and notes for reproducing the benchmark results with the 416 px checkpoint.

Citation

If you use VGGT‑Ω in research, cite the arXiv paper:

@misc{wang2026vggtomega,
  title={VGGT-\$\Omega$},
  author={Jianyuan Wang and Minghao Chen and Shangzhan Zhang and Nikita Karaev and Johannes Sch\"onberger and Patrick Labatut and Piotr Bojanowski and David Novotny and Andrea Vedaldi and Christian Rupprecht},
  year={2026},
  eprint={2605.15195},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2605.15195}
}

Bottom Line

VGGT‑Ω is a state‑of‑the‑art, open‑source vision model that predicts camera geometry and dense depth from images, with optional text‑alignment. The repository supplies pretrained checkpoints, a reproducible training pipeline, and a ready‑to‑run demo, making it a useful starting point for anyone building geometry‑aware AI systems.