PunithVT/ai-avatar-system

What it solves

AvatarAI provides a production-ready platform for creating photorealistic AI avatars that can engage in real-time conversations. It eliminates the gap between static AI chat and immersive digital humans by combining lip-sync video generation, voice cloning, and LLM-driven dialogue into a single, deployable web service.

How it works

The system uses a streaming pipeline to minimize latency. When a user speaks, the audio is processed by Whisper STT, then sent to an LLM (such as Claude, GPT-4o, or a local model via Ollama). The resulting text is split into sentences, which are processed in parallel by a TTS engine (Chatterbox) and a lip-sync engine (MuseTalk). These are delivered as video chunks over WebSockets, allowing the avatar to start speaking before the LLM has finished its entire response.

Who it’s for

It is designed for developers and businesses looking to ship a multi-user AI avatar service. It supports both fully local deployments for privacy and AWS GPU deployments for high-performance, real-time interaction.

Highlights

• Zero-shot voice cloning: Clone a voice using only a 10-second audio sample across 23 languages.
• Real-time lip-sync: Uses MuseTalk V1.5 to achieve 30 FPS on compatible GPUs.
• Barge-in capability: Users can interrupt the avatar mid-reply, and the system cancels the in-flight response instantly.
• Local-first option: Supports 100% local operation using Ollama, Whisper, and local storage.
• Production-grade infra: Includes JWT authentication, rate limiting, PostgreSQL, Redis, and Terraform scripts for AWS deployment.