spite/FaceMeshFaceGeometry

What it solves

FaceMeshFaceGeometry provides a way to convert 2D face landmarks from MediaPipe's Face Landmarker (or TensorFlow.js) into a usable 3D mesh within a three.js environment. It solves the problem of manually mapping raw landmark coordinates to a 3D geometry that aligns with a video feed, while providing essential data like head pose, expression scores, and iris tracking.

How it works

The library creates a BufferGeometry that updates its vertices and normals based on the landmarks provided. It supports several modes of operation:

• Video Space: Vertices are mapped to the pixels of the input video, allowing for easy 2D/3D overlays.
• Metric Mode: Vertices are calculated in centimeters, enabling the use of a perspective camera and providing a Matrix4 for the head pose relative to a canonical face model.
• Texture Mapping: It can generate fixed UV coordinates for masks/makeup or dynamic UVs that follow the video feed for effects like face swapping or video-texturing.
• Smoothing: It uses a One Euro filter to reduce jitter in the landmarks and head pose without introducing significant lag.

Who it’s for

Developers building AR face filters, virtual avatars, 3D face tracking applications, and interactive installations using three.js and MediaPipe.

Highlights

• Integrated Head Pose: Provides a transformation matrix to easily attach 3D objects (like glasses) to the head.
• Real-time Expression Tracking: Accesses blendshape scores (e.g., jawOpen, mouthSmileRight) for interactive animations.
• Iris Tracking: Includes position and radius for both irises to estimate gaze direction.
• Surface Tracking: A track() method allows pinning 3D objects to specific triangles on the face surface.
• Custom Camera Component: Includes <gum-av>, a standalone custom element for simplified webcam access and management.