kocasariumut/FaceAnything

What it is

Face Anything is a research‑grade, feed‑forward neural model that turns any collection of face images (a photo burst, a video, or even a single picture) into a temporally‑consistent 4‑D reconstruction – a 3‑D geometry that moves over time together with dense per‑pixel correspondences. The core idea is canonical facial point prediction: every pixel is mapped to a normalized coordinate in a shared canonical face space, turning tracking and reconstruction into a single dense mapping problem.

What you get out of it

Running the provided run_inference.py on an input sequence produces a suite of visual artefacts:

• Colored point‑cloud videos (orbiting camera) showing the reconstructed face.
• Track videos where each point keeps a consistent colour across frames, visualising dense correspondence.
• Canonical‑coloured renders that colour each point by its canonical coordinate.
• Depth, normal and raw map videos for analysis or downstream use.
• A “grand tour” video that smoothly morphs through all modalities.
• Per‑frame PLY files (geometry, canonical map, tracks) and camera intrinsics/extrinsics (cameras.npz/json).
• A raw_predictions.npz file containing the model’s raw outputs (depth, pose, canonical map, confidence, validity mask).

All of these are written automatically to an output folder; you can select a subset with the --outputs flag.

How to get it running

1. Clone the repo and run the provided install.sh. The script creates a Conda environment, installs PyTorch 2.9 (CUDA 12.8) and the other Python dependencies, and downloads the ~15 GB model checkpoint.
2. GPU requirement – the model needs a CUDA‑capable GPU; it has only been tested on Python 3.11.
3. Checkpoint – the script pulls checkpoint.pt from Google Drive or the Hugging‑Face repo. You can also place the file manually under checkpoints/.
4. Run inference with a single command:

   python run_inference.py --input <path> --output <out_dir>
   

   <path> can be a single image, a folder of images, or a video file.
5. Optional flags let you control resolution, processing mode (all-at-once vs one-by-one), background removal (via Robust Video Matting), which outputs to generate, orbit style, etc.

When you’d want to use it

• Research on 4‑D facial dynamics, expression transfer, or animation where dense, temporally‑consistent geometry is required.
• Content creation for games or AR/VR where a quick turn‑key pipeline from casual footage to a 3‑D face model is useful.
• Dataset generation – the authors release canonical maps for a curated “FaceAnything NeRSemble” dataset (access via a request form).

Limitations & practical notes

• The model is feed‑forward only; it does not perform iterative refinement, so extremely high‑frequency detail may be missing compared to multi‑view SfM pipelines.
• Memory vs detail trade‑off: all-at-once gives smoother temporal consistency but coarser surfaces; one-by-one yields finer geometry at the cost of consistency and higher GPU memory.
• The license is CC‑BY‑NC 4.0, so commercial use is prohibited without permission.

---

Citation

@article{kocasari2026face,
  title={Face Anything: 4D Face Reconstruction from Any Image Sequence},
  author={Kocasari, Umut and Giebenhain, Simon and Shaw, Richard and Nießner, Matthias},
  journal={arXiv preprint arXiv:2604.19702},
  year={2026}
}