TalkingHead (3D)

A browser-side JavaScript class that powers a real-time 3D avatar with speech and lip-sync. It runs in the browser, renders with Three.js/WebGL, and ships with a built-in physics engine for dynamic features like hair. It also maps emojis to facial expressions and supports full-body avatars (GLB format) with Mixamo-style animations (FBX).

What it actually does

• Speaks and lip-syncs: Turns text into spoken audio (via Google Cloud TTS by default) and moves the avatar's mouth in sync, using word-level timestamps. A built-in lip-sync language module covers English, German, French, Brazilian Portuguese, Finnish, and Lithuanian — more languages can be added by writing modules or using an external TTS that provides viseme IDs directly (e.g., Microsoft Azure Speech SDK).
• Adds liveliness: Holds idle head movements, eye contact behaviors, and mood states (neutral, happy, angry, sad, etc.).
• Handles audio playback: Includes methods like speakText for text-to-speech and speakAudio for playing audio files with automatic viseme detection.

What you need to get it running

1. An avatar: A GLB 3D model with a Mixamo-compatible rig and ARKit/Oculus viseme blend shapes (Appendix A in the README explains how to create one).
2. A text-to-speech backend: Either Google Cloud TTS (with a proxy + JWT or an API key), or any external TTS that returns word-level timestamps.
3. The TalkingHead module: Available via the releases page, NPM (@met4citizen/talkinghead), or a CDN import.

What you can customize

The README documents dozens of options, including:

• Voice and speech: language, rate, pitch, volume.
• Camera and view: full, mid, upper, head — plus rotation/pan/zoom permissions.
• Lighting and rendering: ambient, directional, spot light colors and intensities.
• Avatar behavior: eye contact frequency, head movement, initial mood.
• Silence/activity detection: to trigger events when the avatar is listening.

Why it matters

The README lists real-world uses by others, including research projects (human-AI group conversations, digital dating twins, clinical trial recruitment support), live Twitch games, video conferencing, and a recycling advisor app — all built on top of TalkingHead.

---

Key caveat from the README: The built-in Google TTS path recommends using Single Sign-On (SSO) with a proxy rather than exposing an API key in client-side code (which it explicitly warns against in production).