nv-tlabs/vipe

ViPE: Video Pose Engine for Geometric 3D Perception

What it solves

ViPE addresses the challenge of extracting precise 3D spatial information from raw, unconstrained videos. It allows users to automatically annotate camera poses and generate dense depth maps without needing specialized hardware, working across various lens types including pinhole, wide-angle, and 360-degree panoramas.

How it works

ViPE acts as a spatial AI engine that estimates three key components from video footage: camera intrinsics (the internal camera settings), camera motion (the path the camera took), and dense near-metric depth maps (the distance of objects from the camera).

Who it’s for

This tool is designed for developers and researchers working in 3D perception, computer vision, and spatial AI who need to transform raw video into structured 3D data.

Highlights

  • Broad Lens Support: Works with pinhole, wide-angle, and 360-degree panorama footage.
  • Multiple Pipeline Options: Integrates with various pipelines, including Depth-Anything 3 and Lyra.
  • High Performance: Recent updates include CUDA fused kernels and pipeline caching for significant speed-ups.
  • Open Source: Available via PyPI for easy installation.

Related

  • Project
    VIAME/VIAME

    A computer vision toolkit for DIY artificial intelligence, providing tools for object detection, tracking, and annotation across desktop, web, and CLI interfaces.

  • Project
    NVIDIA-ISAAC-ROS/isaac_ros_pose_estimation

    A collection of NVIDIA-accelerated ROS 2 packages for 3D object pose estimation, enabling robots to locate and track both known and novel objects for navigation and manipulation.

  • Project
    VisionDepth/VisionDepth3D

    A comprehensive 3D creation suite that uses AI depth estimation and stereo warping to convert 2D images and videos into VR-ready 3D stereo content.

  • Project
    facebookresearch/vggt-omega

    VGGT-Omega is a computer vision model that estimates camera poses and depth from images to reconstruct 3D scenes.

  • Project
    wkentaro/labelme

    A graphical image annotation tool for creating datasets for computer vision tasks like segmentation and object detection using manual and AI-assisted labeling.