asanchezyali/talking-avatar-with-ai

Talking Avatar with AI

What it is – A small open‑source demo that lets you talk to a 3‑D “digital human” (named Jack). The avatar listens (via Whisper), thinks (via OpenAI’s GPT‑3/ChatGPT), speaks (via Eleven Labs TTS) and moves its lips and facial expression (via Rhubarb Lip‑Sync and pre‑defined animation clips). The whole stack is wired together with a TypeScript/React front‑end and a Node/Yarn monorepo backend.

Core components

• LLM brain – @langchain/openai builds a prompt that forces the model to return a JSON array describing the reply text, a facial expression, and an animation. The response is parsed with zod to guarantee the shape.
• Speech‑to‑text – OpenAI Whisper converts spoken user input to text.
• Text‑to‑speech – Eleven Labs generates high‑quality audio from the LLM’s reply.
• Viseme generation – Rhubarb Lip‑Sync turns the audio into mouth‑cue metadata so the 3‑D face can be lip‑synced.
• Front‑end – A React app (using Tailwind CSS and React‑Three‑Fiber) renders the avatar, applies the chosen animation/expression, and plays the audio.

How it works (high‑level flow)

1. User input – either typed text or an audio recording.
2. Transcription (audio only) – Whisper → plain text.
3. LLM call – the text is sent to OpenAI’s GPT model with a custom prompt that asks for a JSON payload containing text, facialExpression, and animation.
4. TTS – The text field is sent to Eleven Labs, which returns an audio file.
5. Lip‑sync – The audio file is fed to Rhubarb, which outputs viseme timestamps (mouthCues).
6. Render – The front‑end reads the JSON, switches the avatar’s facial expression and animation, plays the audio, and drives the lip‑sync using the viseme data.

Getting started

1. Prerequisites – OpenAI, Eleven Labs, and Rhubarb accounts (paid tiers recommended), ffmpeg, and a recent Node/Yarn environment.
2. Clone & install

   git clone git@github.com:asanchezyali/talking-avatar-with-ai.git
   cd digital-human
   yarn   # installs the monorepo packages
   

3. Configure – create /apps/backend/.env with your OpenAI and Eleven Labs keys and IDs.
4. Run – yarn dev and open http://localhost:5173/.

What you’ll see – a web page showing a 3‑D avatar that reacts to your spoken or typed queries with appropriate facial expressions (smile, sad, angry, surprised, funnyFace, default) and body animations (Idle, TalkingOne, TalkingThree, etc.). The avatar’s voice comes from Eleven Labs, and its lips move in sync thanks to Rhubarb.

Limitations / notes

• The demo relies on external paid APIs; free tiers may hit rate limits and cause the avatar to stall.
• Only a fixed set of expressions and animations are supported; extending them requires editing the prompt and adding new animation assets.
• The repository is a proof‑of‑concept rather than a production‑ready framework – error handling, scalability, and UI polish are minimal.

Useful links

• Detailed tutorial: https://monadical.com/posts/build-a-digital-human-with-large-language-models.html
• Discord support channel: https://discord.gg/gJ3vCgSWeh
• Rhubarb Lip‑Sync repo: https://github.com/DanielSWolf/rhubarb-lip-sync

---

All information above is taken directly from the repository’s README; no additional features have been assumed.