met4citizen/TalkingHead

TalkingHead (3D) – What It Is

TalkingHead is a JavaScript library that lets you put a 3‑D avatar (a GLB model with a Mixamo‑compatible rig) into a web page and make it talk, gesture, and show facial expressions in real‑time. The avatar can lip‑sync to spoken audio, display emojis as facial poses, and be driven by various AI services (OpenAI, Google TTS, Whisper, etc.). It is built on Three.js for rendering and works entirely in the browser – you can load it from a CDN or install it via npm.

---

Core Features

Feature	What It Does
Real‑time lip‑sync	Generates visemes from text‑to‑speech services (Google Cloud TTS by default) or from audio‑driven modules like HeadAudio. Supports several languages out of the box (en, fi, de, fr, lt) and can be extended with custom modules.
Full‑body avatars	Accepts GLB files with Mixamo rigs, ARKit/Oculus blend‑shape definitions, and dynamic‑bone physics for hair and clothing.
Emoji‑to‑expression mapping	Built‑in emoji dictionary turns standard emojis into facial blend‑shape values.
Pluggable TTS back‑ends	Works with Google Cloud TTS, ElevenLabs, Azure Speech, or any service that can return word‑level timestamps or viseme IDs.
LLM‑driven control	Example demos show how OpenAI function calling can drive avatar gestures and speech.
WebRTC‑ready	A companion project (HeadAudio) provides low‑latency audio worklet for speech‑to‑speech over WebRTC.
Add‑on ecosystem	Companion modules – HeadTTS (browser‑only neural TTS), HeadAudio (audio‑driven viseme detection), and MotionEngine (advanced gesture library with LLM‑semantic control).
Configurable rendering	Camera, lighting, and avatar‑level options are exposed via a large options object; you can enable/disable rotation, pan, zoom, change moods, etc.

---

Typical Use Cases (as shown in the README)

• Human‑AI group conversations – used in the DialogLab toolkit for multi‑person chat simulations.
• Realtime AI speech over WebRTC – the HeadAudio demo streams speech‑to‑speech with lip‑sync.
• Video‑conferencing avatars – a demo with an avatar named Olivia that adds transcription and AI‑generated replies.
• Fully in‑browser AI assistants – the EdgeSpeaker site runs everything (LLM, TTS, lip‑sync) locally with WebGPU, no external API keys needed.
• Interactive 3‑D experiences – recycling advisor, Twitch adventure world, quantum‑physics teaching, AI‑driven dating‑profile avatars, etc.

---

Getting Started (quick‑start)

1. Include the library – either via npm (npm i @met4citizen/talkinghead) or directly from a CDN:

   <script type="importmap">
   {"imports":{"talkinghead":"https://cdn.jsdelivr.net/gh/met4citizen/TalkingHead@1.7/modules/talkinghead.mjs"}}
   </script>
   

2. Create a TalkingHead instance:

   import { TalkingHead } from "talkinghead";
   const head = new TalkingHead(document.getElementById('avatar'), {
     ttsEndpoint: "/gtts/",   // proxy to Google TTS
     jwtGet: myJwtFetcher,      // optional SSO token getter
     lipsyncModules: ["en","fi"],
   });
   

3. Load an avatar (GLB file with proper rig):

   await head.showAvatar({
     url: "./avatars/avaturn.glb",
     body: "F",
     avatarMood: "neutral",
     ttsLang: "en-GB",
     lipsyncLang: "en",
   });
   

4. Make it speak:

   head.speakText("Hello, world!");
   

The README also provides a minimal HTML example that works with just a Google TTS API key.

---

Extending / Customising

• Add your own TTS – implement a service that returns word‑level timestamps or viseme IDs and point ttsEndpoint (or use the ttsApikey shortcut for direct Google calls).
• New languages – create a lip‑sync language module and list it in lipsyncModules.
• Dynamic bones & physics – define modelDynamicBones in the avatar options (see Appendix E).
• Gesture library – plug in MotionEngine for semantic LLM‑driven gestures.
• Avatar‑only mode – set avatarOnly: true to get just the armature object for integration into an existing Three.js scene.

---

Who Might Use It?

• Researchers building embodied AI agents, conversational UI studies, or multimodal interaction experiments.
• Developers creating virtual presenters, AI‑driven customer‑service avatars, or game‑style NPCs.
• Educators / hobbyists who want a quick way to add a talking 3‑D character to a web demo without dealing with heavy backend infrastructure.

---

License & Availability

The library is released under an open‑source license (the README links to GitHub releases and an npm package). All code, demo videos, and companion modules are publicly accessible.

---

Bottom line: TalkingHead is a fully‑featured, browser‑based 3‑D avatar framework that bridges graphics, speech synthesis, and LLM control, making it a practical tool for anyone building interactive, AI‑powered virtual characters.