ShenhanQian/VHAP

A complete head tracking pipeline from videos to NeRF/3DGS-ready datasets.

What it solves

VHAP addresses the challenge of aligning 3D human head models to video sequences, particularly in regions where pre-defined landmarks (like those for eyes or nose) are unavailable, such as the hair, ears, neck, and shoulders.

How it works

It uses a photometric optimization pipeline based on differentiable mesh rasterization. The system employs a perturbation mechanism that adaptively extracts and injects regional appearance priors during rendering. This allows the model to align specific regions based on their appearance consistency rather than relying solely on landmarks. It supports both monocular videos and multi-view datasets and integrates with the FLAME head model for shape and pose estimation.

Who it’s for

This tool is designed for researchers and developers working on 3D head tracking, digital human creation, and NeRF/3DGS-style applications, such as creating photorealistic head avatars.

Highlights

  • Adaptive Appearance Priors: Enables alignment of non-landmark regions like hair and ears.
  • Versatile Input: Supports both single-camera (monocular) and multi-camera (multi-view) video sequences.
  • Differentiable Rasterization: Utilizes a photometric optimization pipeline for high-precision alignment.
  • Avatar Integration: Exported tracking results can be used directly to generate GaussianAvatars.

Related

  • Project
    hysts/pytorch_mpiigaze

    A PyTorch implementation of gaze estimation methods from the MPIIGaze and MPIIFaceGaze papers for training and evaluation of appearance-based gaze tracking.

  • Project
    nipreps/mriqc

    An open-source tool that extracts no-reference image quality metrics from structural, functional, and diffusion MRI data for automated quality assessment.

  • Project
    beixiaocai/rebucca

    A multi-channel video analysis platform that combines YOLO detection and LLM verification to provide structured alarms for security and monitoring.

  • Project
    NVlabs/SpatialClaw

    SpatialClaw is a training‑free, code‑as‑action framework for spatial‑reasoning agents. A VLM writes Python cells that run in a persistent Jupyter kernel pre‑loaded with perception tools (SAM‑3, Depth‑Anything‑3) and scientific libraries. The system loops through planning, code generation, safety‑checked execution, and observation until the agent returns an answer. Evaluated on 20 benchmarks, it reaches 59.9 % average accuracy, beating prior agents by 11.2 points, using the same prompt and tools across six VLM back‑ends. The repo provides full runtime, benchmark loaders, installation scripts, and SLURM support.