datascale-ai/opentalking

OpenTalking: An industrial-grade open-source AI digital human framework that supports real-time conversation, private deployment, and pluggable models.

What it solves

OpenTalking provides a complete, open-source orchestration framework for creating real-time digital-human conversations. It eliminates the complexity of manually stitching together separate models for speech-to-text (STT), large language models (LLMs), text-to-speech (TTS), and video rendering to create an interactive, talking avatar.

How it works

The project acts as a production stack that manages the entire conversational pipeline: frontend interaction, session state, LLM responses, voice selection, interruption control, and WebRTC audio/video playback. It supports multiple deployment paths, from a lightweight "mock" mode for API validation to high-quality remote inference using OmniRT. It integrates various digital-human driver models (such as QuickTalk, Wav2Lip, and FlashTalk) to animate avatars based on audio or text input.

Who it’s for

It is designed for developers and creators building digital-human products for scenarios like healthcare guidance, live commerce, virtual companions, and news anchors, as well as those wanting to experiment with audio-driven video creation and video cloning.

Highlights

  • Pluggable Backends: Supports a wide range of LLM, STT, and TTS providers, including OpenAI-compatible interfaces.
  • Flexible Deployment: Offers paths for CPU-only trials, consumer-GPU local setups (RTX 3090/4090), and high-quality remote GPU/NPU clusters.
  • Comprehensive Toolset: Includes a WebUI for managing avatars, voices, and model configurations, along with support for knowledge bases and character memory.
  • Diverse Rendering Models: Integrates multiple drivers including QuickTalk for speed, Wav2Lip for accessibility, and FasterLivePortrait for video cloning and high-quality creation.

Related

  • Project
    lipku/LiveTalking

    A real-time interactive streaming engine for digital humans that synchronizes audio and video for lifelike AI-driven dialogue.

  • Project
    bigsk1/voice-chat-ai

    A voice-driven AI interface that allows users to interact with diverse AI characters using a variety of LLM and TTS providers, supporting real-time conversations and local voice cloning.

  • Project
    kyutai-labs/unmute

    A system that enables text LLMs to listen and speak by wrapping them in low-latency speech-to-text and text-to-speech models.

  • Project
    rsxdalv/TTS-WebUI

    A unified web interface and manager for a wide variety of open-source text-to-speech, audio generation, and audio conversion AI models.

  • Project
    TEN-framework/ten-framework

    An open-source framework for building real-time multimodal conversational AI agents with low-latency audio, text, and visual capabilities.