BQ-Wang0511/TalkLikeYou-Streaming-RealTime-Demo

TalkLikeYou – Real‑time Talking‑Head Demo

What it does – A browser‑based demo that turns a static portrait (or a video) into a talking‑head that lip‑syncs to uploaded audio, a live microphone, or speech generated by a chat‑LLM. It can also dub an existing video, keeping the original pose and timing while replacing the mouth motion, and it can act as a chat‑agent that streams LLM replies, TTS audio, and avatar animation together.

Key capabilities

• Audio mode – Upload an image avatar, then feed it an audio file or live mic input; the system generates lip motion in real time.
• Video dubbing – Provide a source video and a new audio track; the demo keeps the original head pose and replaces only the speech‑related lip motion.
• Chat agent – Connect to any OpenAI‑compatible chat, STT, and TTS services. Typed or spoken user input is sent to the LLM, the reply is streamed back as text, synthesized speech, and synchronized avatar animation.
• Preset speaking styles – “habit IDs” (e.g., 192, 166, 202) let you switch between different lip‑generation styles without needing a reference video.
• Real‑time performance – On an NVIDIA RTX 3090 the system runs at >30 FPS while using <4 GB GPU memory, delivering high‑clarity facial details.

How it works (high‑level)

1. Avatar registration – The uploaded image/video is processed to detect the face, crop it, and cache portrait features (including a neutral mouth motion shell). This step is done once per avatar.
2. Streaming renderer – Audio is chunked (default 1 s) and fed to a habit‑conditioned lip‑motion generator; a Flow‑Matching pose branch (or Ditto‑audio‑driven pose) supplies head motion. The generated frames are composited back onto the original source and streamed to the browser.
3. Chat integration – The server forwards user text or recorded speech to the configured LLM and TTS endpoints. As each token/segment arrives, the corresponding speech audio and avatar frames are generated and displayed.

Installation

# Create the conda environment (Python 3.10, CUDA 12.1, cuDNN 9)
conda env create -f environment.yaml
conda activate talklikeyou-demo
# Or install PyTorch manually and then:
pip install -r requirements.txt

Requirements: Linux, NVIDIA GPU, FFmpeg.

Running the demo

cd TalkLikeYou-Streaming-RealTime-Demo
CUDA_VISIBLE_DEVICES=0 python app.py --host 0.0.0.0 --port 5070
# Open http://127.0.0.1:5070 in a browser

• Use the UI to select Audio, Video dubbing, or Chat agent.
• Upload an image (Audio/Chat) or video (Video dubbing) and register the avatar.
• For Audio, choose an audio file or enable Live microphone and click Start Streaming.
• For Video dubbing, upload the source video, an audio file, and start streaming.
• For Chat agent, configure LLM/STT/TTS endpoints via environment variables (or a .env file) and interact through the chat box.

Configuration highlights

• Habit IDs (Lip person ID) control speaking style; recommended values are 192, 166, 202.
• Guidance strengths (Lip guidance, Pose guidance) adjust how strongly the model follows the selected habit.
• Sampling steps – Real‑time setting is 1 step; more steps increase latency.
• Chunk sizes – Motion chunk (frames) and Audio chunk (seconds) trade off latency vs. throughput.
• Ditto pose – When enabled, uses Ditto’s audio‑driven pose generator for image avatars; disabling it lets you select a pose habit.
• Chat agent – Set TALKLIKEYOU_LLM_BASE_URL, TALKLIKEYOU_LLM_API_KEY, etc., to point to any OpenAI‑compatible service; similarly for STT and TTS.

Model checkpoints

• Download the required checkpoints from the Hugging Face repo doubi-killer/TalkLikeYou and place them under a local checkpoints/ directory as described in the README (audio encoder, lip_motion.pt, pose_motion.pt, and runtime assets).
• The demo ships with a runtime_config.pkl that points to these local paths.

Licensing

• Demo code: MIT License.
• Third‑party components (Ditto, LivePortrait) retain their original licenses (see THIRD_PARTY_NOTICES.md).

Who might use this

• Researchers exploring real‑time lip‑sync and avatar animation.
• Developers building interactive virtual agents, streaming avatars, or video dubbing tools.
• Creators who want a lightweight, locally‑run talking‑head demo without cloud inference.

Contact – Questions can be directed to wangbaiqin0511@gmail.com.