lipku/LiveTalking

Real time interactive streaming digital human

What it solves

LiveTalking is a real-time interactive streaming engine for digital humans. It solves the problem of creating lifelike, synchronized audio-visual dialogue systems where a virtual avatar can speak in response to text or voice input with realistic lip-syncing, allowing for seamless interaction in live streaming, customer service, and education.

How it works

The system follows a pipeline: User Input (Text/Audio) $\rightarrow$ LLM (for dialogue generation) $\rightarrow$ TTS (Text-to-Speech) $\rightarrow$ Digital Human Model (for lip-syncing) $\rightarrow$ Streaming Output.

It utilizes a modular architecture:

  • API Layer: Manages sessions and handles requests via /human and /humanaudio endpoints.
  • Logic Layer: Integrates LLMs (like Qwen) and various TTS engines (EdgeTTS, GPT-SoVITS, CosyVoice) to generate speech.
  • Rendering Layer: Uses deep learning models such as Wav2Lip, MuseTalk, and ER-NeRF to generate lip-sync frames based on audio features.
  • Streaming Layer: Delivers the video stream via WebRTC (for low latency), RTMP (for broadcasting), or as a virtual camera.

Who it’s for

This project is designed for developers and businesses looking to implement AI avatars for:

  • Virtual Streamers: 24/7 unmanned live streaming with automated scripts. -- AI Customer Service: Real-time voice interaction with corporate knowledge bases. -- Online Education: Creating digital teacher clones for real-time or recorded lessons.
  • Smart Voice Assistants: Integrating digital humans into apps or smart speakers.
  • Content Creators: Batch producing short videos without needing a physical shoot.

Highlights

  • Multi-Model Support: Compatible with ernerf, musetalk, wav2lip, and Ultralight-Digital-Human.
  • Low Latency: Supports WebRTC for real-time browser-based interaction.
  • Voice Cloning: Integrated support for voice cloning to personalize the avatar's voice.
  • Interruptible Dialogue: Allows users to interrupt the digital human while it is speaking.
  • Flexible Output: Supports WebRTC, RTMP, and virtual camera output.
  • Action Orchestration: Ability to play custom videos when the avatar is not speaking.
  • Custom Avatars: Support for creating and custom digital human images.

Related

  • Project
    datascale-ai/opentalking

    An open-source orchestration framework for real-time digital-human conversations, integrating LLMs, TTS, STT, and video rendering via WebRTC.

  • Project
    PunithVT/ai-avatar-system

    A production-ready platform for creating photorealistic AI avatars with real-time lip-sync, zero-shot voice cloning, and multi-LLM support.

  • Project
    ardha27/AI-Waifu-Vtuber

    An AI-powered VTuber assistant that integrates speech recognition, LLMs, and text-to-speech to create an interactive virtual character for live streaming and personal use.

  • Project
    Soul-AILab/SoulX-FlashTalk

    SoulX-FlashTalk is a system for real-time, infinite streaming of audio-driven avatars using self-correcting bidirectional distillation to create continuous talking head animations.

  • Project
    fudan-generative-vision/Hallo-Live

    Hallo-Live is a real-time text-driven framework that generates synchronized audio and video for digital avatars using a causal dual-stream Diffusion Transformer.