emilianavt/OpenSeeFace

Robust realtime face and facial landmark tracking on CPU with Unity integration

OpenSeeFace – Real‑time facial‑landmark tracker for avatar animation

What it is – OpenSeeFace is a lightweight, cross‑platform library that detects a set of 66 facial landmarks (plus gaze and blink information) from a webcam or video stream. The core model is a MobileNetV3‑based neural network exported to ONNX, so it can run fast on CPU using onnxruntime (30‑60 fps on a single core). It is not a full avatar‑puppeteering app; instead it streams the tracking data over UDP so other programs (e.g., Unity, Godot, VSeeFace, VTube Studio) can animate 3D/2D avatars.

Why it matters – Real‑time avatar puppeteering needs stable, low‑latency facial landmarks that work under noisy lighting, low resolution, and a wide range of head poses. OpenSeeFace trades a bit of pinpoint accuracy for robustness and speed, making it suitable for live‑streaming, VTubing, and other interactive media where CPU resources are limited.

Key features

Feature Details
Facial landmark model 66 points (iBUG‑68‑like) plus quasi‑3D contour points; includes eye‑open/close, gaze, and blink cues.
Model variants Five pre‑trained models (‑1 … 3) from ultra‑fast/low‑accuracy to slower/high‑accuracy (≈213 fps → 44 fps on one CPU core).
ONNX Runtime inference Runs on Windows, Linux, macOS without a GPU; a pre‑built facetracker.exe is provided for Windows.
UDP output Sends per‑frame landmark data as a binary packet; can be received on a separate machine to keep the capture PC isolated.
Unity integration OpenSee, OpenSeeShowPoints, OpenSeeLauncher, OpenSeeIKTarget, and OpenSeeExpression components handle packet reception, visualization, IK driving, and per‑user expression training.
Cross‑language support The tracker is a Python 3 script (facetracker.py) that can be run directly or via the compiled executable; dependencies are minimal (onnxruntime, OpenCV, Pillow, NumPy).
Expression classifier Optional SVM‑based expression detector that can be calibrated per user and saved/loaded.
Robustness Works with partial occlusion, glasses, low light, and a wide head‑pose range; face detection uses a custom heat‑map regression model or RetinaFace.

Typical workflow

  1. Start the tracker – Run facetracker.exe (Windows) or python facetracker.py (any OS). Choose a camera or video file, select a model (--model 0‑3), and optionally enable --visualize to see the landmarks locally.
  2. Stream data – The script sends UDP packets containing the landmark coordinates (and optional gaze/blink flags).
  3. Consume in Unity – Add the OpenSee component to a GameObject; it receives the packets in a background thread and exposes them via trackingData. Use OpenSeeShowPoints to visualise the points, or feed the data to a VRM/Live2D avatar controller.
  4. Optional expression training – Add OpenSeeExpression, record a few examples per expression, train the SVM, and then query IsExpressionActive("smile") in your avatar logic.

Strengths

  • CPU‑only real‑time performance – No GPU required; suitable for low‑end laptops or streaming rigs.
  • Robust under adverse conditions – Handles low light, noise, and partial occlusion better than many comparable solutions (e.g., MediaPipe) for avatar use‑cases.
  • Modular design – UDP streaming lets you separate capture from rendering, reducing latency spikes and protecting camera privacy.
  • Multiple model sizes – You can pick a faster model for many faces or a higher‑quality one for a single‑person stream.
  • Open source & permissive license – BSD‑2‑clause, easy to embed in commercial projects.

Limitations

  • Eye‑region accuracy – Landmark positions around the eyes are less precise than some dedicated eye‑trackers; blinking detection works but fine‑grained gaze may be jittery.
  • CPU load – Tracking a single face at 30 fps can occupy nearly one full CPU core; higher frame rates or multiple faces increase usage proportionally.
  • No built‑in avatar renderer – You must provide your own Unity/Unreal/Godot integration (sample projects are supplied but not a full end‑user app).
  • Linux onnxruntime loading quirks – Some users need to adjust the executable stack (execstack -c …) to get the library to load.

Getting started quickly

# Install dependencies (recommended via uv)
uv sync --locked          # creates a virtual environment
uv run facetracker.py --visualize 3 -c 0   # webcam, model 3, show landmarks

In Unity:

  1. Create an empty GameObject.
  2. Add OpenSee and OpenSeeShowPoints components.
  3. Press Play – the tracker will start (or start it manually with OpenSeeLauncher).

License

The code and pretrained models are released under the BSD 2‑clause license. Third‑party library licenses are included in the Licenses folder of the release binaries.

Where to find more

Related

  • Project
    bytedance/Protenix

    An open-source biomolecular structure prediction framework that provides high-accuracy 3D modeling of proteins and RNA, designed to be an accessible alternative to AlphaFold3.

  • Project
    opendatalab/OmniDocBench

    A comprehensive benchmark for evaluating document parsing in real-world scenarios, featuring rich annotations for layout detection, OCR, table, and formula recognition.

  • Project
    Farama-Foundation/ViZDoom

    A Doom-based AI research platform for developing and testing bots that play the game using only visual information, primarily for reinforcement learning research.

  • Project
    DanBloomberg/leptonica

    A comprehensive ANSI C image processing and analysis library used by projects like Tesseract and OpenCV to handle document and natural image manipulation.