yizhi-chengzi/video-ai-talking

video‑ai‑talking – AI‑driven “real‑person” talking‑head video generator

What it is

• A locally‑run web app that turns a silent video of a person’s face into a full‑length talking‑head video. You supply the script (or let the tool generate one with DeepSeek), the app creates synthetic speech, lip‑syncs it to the face using Alibaba Bailian VideoRetalk, and stitches everything together with optional background footage and subtitles via FFmpeg.

Why it matters

• Lets creators produce “talking‑head” videos without having to record voice‑overs or manually edit lip‑sync. All processing stays on your machine – no cloud server stores your keys or media.

Key components

Component	Role	Provider
Text‑to‑speech	Generates the spoken audio	Alibaba Bailian CosyVoice or Volcengine (ByteDance) TTS
Lip‑sync	Aligns audio to facial movements	Alibaba Bailian VideoRetalk
Script generation (optional)	Writes the narration from a topic	DeepSeek LLM
Video assembly	Combines face video, generated speech, optional BGM and extra footage, adds subtitles/titles	FFmpeg (local)

What you need

• Node 20+ and a working FFmpeg/ffprobe installation.
• API credentials for the services you plan to use:
  • Bailian VideoRetalk (required for lip‑sync)
  • Bailian CosyVoice or Volcengine TTS (speech)
  • DeepSeek (optional, for AI‑generated script)
• A short, front‑facing video of a single person (no speech needed).

How to get started

# clone and install
git clone https://github.com/yizhi-chengzi/video-ai-talking.git
cd video-ai-talking
cp .env.example .env
npm install

# development mode (opens at http://127.0.0.1:5175)
npm run dev

• Open the web UI, fill the Configuration panel with your API keys, and test the connections.
• Drag‑and‑drop your “talking‑head” clip, add any background footage you want, choose a TTS voice, and either write the script yourself or let the AI generate one.
• Pick a subtitle/skin style, then click Start Generation. The pipeline runs: TTS → VideoRetalk → FFmpeg → final MP4.
• Finished videos appear in the Library panel for preview, download, or deletion.

Local data handling

• API keys are stored only in the browser’s localStorage (never sent to a remote server).
• All intermediate files (JSON task description, raw audio, lip‑sync video, final MP4) live under the project’s data/ folder.
• Temporary uploads required by VideoRetalk are stored on Alibaba’s OSS for ~48 h and then deleted.

Typical use‑case flow

1. Configure – enter Bailian VideoRetalk key; pick a TTS provider and enter its credentials.
2. Load source video – a clear, front‑facing clip of the presenter.
3. Add assets – optional BGM or full‑screen cut‑in footage.
4. Create script – type it manually or let DeepSeek generate one (choose length, topic).
5. Select skin – choose subtitle style, title visibility, etc.
6. Generate – the app produces the spoken audio, lip‑sync video, and final MP4 in one go.
7. Review – watch or download from the Library; re‑generate if needed.

Development & testing

• npm test runs unit tests with all external services mocked.
• npm run typecheck checks TypeScript types.
• Set VAT_MOCK=1 to mock TTS, VideoRetalk, and DeepSeek for offline experimentation.

Safety & licensing

• MIT‑licensed, but see the bundled DISCLAIMER.md, SECURITY.md, and privacy.md for usage cautions (e.g., facial data is uploaded temporarily to Bailian for lip‑sync).
• The demo videos shown in the README are from free‑to‑use stock libraries and are not actual users of the tool.

---

Bottom line: video‑ai‑talking is a practical, self‑hosted tool for generating AI‑assisted talking‑head videos, combining modern TTS, lip‑sync, and video compositing while keeping all data on your own computer.