PunithVT/ai-avatar-system

🎭 AI Avatar / digital human platform — upload a photo, clone a voice, talk to any face in real time with lip-sync video. Open-source, self-hosted. Claude · Whisper · Chatterbox · MuseTalk.

What it solves

AvatarAI provides a production-ready platform for creating photorealistic AI avatars that can engage in real-time conversations. It eliminates the gap between static AI chat and immersive digital humans by integrating speech-to-text, LLMs, voice cloning, and lip-sync video generation into a single, low-latency pipeline.

How it works

The system uses a streaming pipeline to minimize delay. When a user speaks, Whisper STT converts audio to text, which is sent to an LLM (Claude, GPT-4o, or local models via Ollama). The resulting text is split into sentences and processed in parallel: each sentence is converted to speech via Chatterbox TTS (supporting zero-shot voice cloning) and then animated using the MuseTalk V1.5 engine to generate lip-sync video chunks. These chunks are streamed to the browser via WebSockets, allowing the avatar to start speaking before the LLM has finished its entire response.

Who it’s for

It is designed for developers and businesses looking to deploy a multi-user web service for digital humans, rather than just a research demo. It is suitable for those who want to run the system 100% locally for privacy or deploy it on AWS GPU instances for high-performance, real-time interaction.

Highlights

  • Real-time Lip-Sync: Uses MuseTalk V1.5 to achieve up to 30 FPS on GPU.
  • Zero-Shot Voice Cloning: Clones a voice from a 10-second audio sample in 23 different languages.
  • Barge-In Capability: Allows users to interrupt the avatar mid-reply for more natural conversations.
  • Multi-LLM Support: Compatible with Claude, GPT-4o, and local engines like Ollama, vLLM, or LM Studio.
  • Production-Grade Infrastructure: Includes JWT authentication, rate limiting, PostgreSQL database, and AWS Terraform scripts for deployment.

Related

  • Project
    lipku/LiveTalking

    A real-time interactive streaming engine for digital humans that synchronizes audio and video for lifelike virtual avatars driven by LLMs and TTS.

  • Project
    bigsk1/voice-chat-ai

    A voice-driven AI interface that allows users to interact with diverse AI characters using a variety of LLM and TTS providers, supporting real-time conversations and local voice cloning.

  • Project
    uezo/aiavatarkit

    A modular framework for building low-latency conversational AI avatars with integrated VAD, STT, LLM, and TTS pipelines across multiple channels.

  • Project
    datascale-ai/opentalking

    An open-source orchestration framework for real-time digital-human conversations, integrating LLMs, TTS, STT, and video rendering via WebRTC.

  • Project
    Saganaki22/ComfyUI-OmniVoice-TTS

    ComfyUI nodes for OmniVoice, providing zero-shot multilingual text-to-speech with voice cloning, voice design, and multi-speaker dialogue support for over 600 languages.