nv-tlabs/vipe

ViPE: Video Pose Engine for Geometric 3D Perception

What it solves

ViPE is a spatial AI tool designed to extract geometric 3D information from raw videos. It solves the problem of manually annotating camera poses and dense depth maps, automating the process for a wide variety of video types, including standard pinhole, wide-angle, and 360-degree panorama footage.

How it works

ViPE estimates camera intrinsics, camera motion (poses), and dense near-metric depth maps from unconstrained raw video. For standard videos, it uses a fixed-size keyframe buffer. For arbitrarily long videos, it employs a pose_only_long pipeline that uses a sliding keyframe window and a trajectory ledger to keep GPU and CPU memory usage bounded and constant regardless of the video length.

Who it’s for

This tool is intended for developers and researchers working with 3D perception, spatial AI, and video analysis who need accurate camera trajectories and depth maps from raw footage.

Highlights

  • Versatile Video Support: Handles pinhole, wide-angle, and 360-degree panorama videos.
  • Memory Efficient: Includes a specialized pipeline for long-sequence videos to prevent memory growth.
  • High Performance: Optimized with CUDA fused kernels and pipeline caching for significant speed-ups.
  • Flexible Depth Integration: Integrates with various depth estimation models, including Depth-Anything 3 and MoGe v2.

Related

  • Project
    VIAME/VIAME

    VIAME is a BSD‑licensed, open‑source computer‑vision toolkit that provides a modular C++/Python pipeline framework, a large set of ready‑made detection/tracking/measurement algorithms, and both desktop and web GUIs. It can be installed via pre‑built binaries, Docker, or pip, and supports GPU acceleration through CUDA, PyTorch, ONNX, TensorRT, etc. Ideal for researchers and developers building video analytics pipelines.

  • Project
    VisionDepth/VisionDepth3D

    VisionDepth 3D is a Windows desktop app that converts 2‑D images or videos into stereoscopic 3‑D using AI depth models, depth‑map blending, RIFE frame‑interpolation, and AI super‑resolution. It offers a free tier (limited length, 1080p, watermarked) and a one‑time $59.99 Pro tier (no limits, 4K+, batch processing, advanced controls). Installation is via an Itch.io “Setup Hub” that provides CUDA (NVIDIA) or DirectML (AMD/Intel) builds. The software is proprietary, but the AI models it downloads are open‑source under their own licenses.

  • Project
    NVIDIA-ISAAC-ROS/isaac_ros_pose_estimation

    A collection of NVIDIA-accelerated ROS 2 packages for 3D object pose estimation, enabling robots to locate and track known and novel objects for navigation and manipulation.

  • Project
    wkentaro/labelme

    A graphical image annotation tool written in Python and Qt that enables the creation of datasets for computer vision tasks like segmentation and detection.

  • Project
    stereolabs/zed-sdk

    A cross-platform library for ZED cameras that provides real-time 3D spatial perception, including depth sensing, object detection, and body tracking.