PunithVT/ai-avatar-system

🎭 AI Avatar / digital human platform — upload a photo, clone a voice, talk to any face in real time with lip-sync video. Open-source, self-hosted. Claude · Whisper · Chatterbox · MuseTalk.

🎭 AvatarAI – Real‑time AI avatar platform

What it is – An open‑source, production‑ready stack that lets you turn any portrait photo into a talking digital human. Upload a face, give a short voice sample, and you can have a live, lip‑synced conversation with the avatar. The system streams LLM‑generated text, runs it through a TTS engine, feeds the audio to a lip‑sync model, and pushes video chunks back to the browser over WebSocket, all in a few seconds.

Core capabilities (as described in the README)

Capability	How it works	Key details
Zero‑shot voice cloning	Uses Chatterbox Multilingual; 10‑60 s of audio is enough to create a synthetic voice in 23 languages.	No external API required; optional fallback chain (Chatterbox → edge‑tts → gTTS).
Photorealistic lip‑sync	MuseTalk V1.5 runs as a persistent GPU worker, generating 256×256 mouth frames at ~30 FPS on an A10G GPU.	Can fall back to CPU‑based FFmpeg; optional GFPGAN face‑restoration for sharper output.
Streaming LLM pipeline	Whisper (STT) → Claude / GPT‑4o / local LLM (Ollama, vLLM, LM Studio) → sentence‑wise TTS → MuseTalk → WebSocket video chunks.	Tokens are streamed from the LLM, so the first video chunk appears after 2–4 s on AWS GPU.
Barge‑in / interruption	Front‑end detects voice activity; sending a stop signal cancels the in‑flight turn within milliseconds.
Multi‑LLM support	Cloud APIs (Claude, GPT‑4o) or free local models via Ollama/vLLM/LM Studio.
Auth & persistence	JWT + http‑only cookies, PostgreSQL (Alembic migrations) for users, avatars, conversation history, Redis for rate‑limiting & Celery task queue.
Observability	Prometheus metrics, Celery Flower UI, optional Sentry integration, structured logging.
Deployability	Docker‑Compose for local dev (CPU‑only) and GPU‑enabled production; one‑command EC2 deployment script; Terraform for managed ECS + RDS + ElastiCache + CloudFront.
Extensible avatar engine	`AVATAR_ENGINE` env var – `simple` (no lip‑sync), `musetalk` (default), optional `liveavatar` (high‑fidelity but needs 48‑80 GB VRAM).

Architecture at a glance

Browser (Next.js)                     FastAPI backend (Python)
 ├─ Avatar Studio (upload)                ├─ WebSocket manager (sentence streaming)
 ├─ Voice Studio (clone)                  ├─ Whisper STT (faster‑whisper, CUDA)
 └─ Chat UI (WebSocket video)             ├─ LLM adapters (Claude, GPT‑4o, Ollama, vLLM)
                                          ├─ TTS (XTTS v2, Chatterbox, edge‑tts)
                                          └─ MuseTalk worker (GPU, persistent)

Data stores: PostgreSQL (users, avatars, convo), Redis (rate‑limit, Celery broker),
local filesystem or S3 (uploads, generated video), Prometheus/Sentry for metrics.

Quick‑start (Docker‑only, CPU dev)

git clone https://github.com/PunithVT/ai-avatar-system.git
cd ai-avatar-system
cp .env.example .env               # add your Anthropic or OpenAI key
docker compose up -d               # builds backend, frontend, redis, postgres

Frontend: http://localhost:3000
Backend API: http://localhost:8000 (Swagger at /docs)
Celery Flower (optional): http://localhost:5555

To try a ready‑made avatar:

backend/venv/bin/python scripts/seed_demo.py --with-voices

GPU / AWS deployment

Recommended instance: g5.xlarge (A10G, 24 GB VRAM) – ~30 FPS lip‑sync.
One‑command EC2 script (scripts/deploy-aws.sh) installs Docker, pulls the images, downloads MuseTalk weights (~9 GB), and starts the stack with GPU passthrough.
For a fully managed setup, run the Terraform files in infrastructure/ (creates ECS service, RDS, ElastiCache, S3 + CloudFront) and then bash deploy.sh production.

Roadmap highlights (from README)

RAG integration – add vector‑store retrieval to augment LLM responses.
Higher‑fidelity video engines – better than MuseTalk for offline rendering.
Improved face‑restoration pipeline – smoother integration of GFPGAN.
Multi‑avatar orchestration – support for simultaneous conversations with different avatars.

Who might use this?

Developers building a commercial AI‑assistant or virtual‑host service who need a ready‑made, self‑hosted stack.
Researchers experimenting with real‑time multimodal pipelines (STT → LLM → TTS → lip‑sync) without wiring together many separate repos.
Content creators looking for a locally‑run avatar that can be customized with their own face and voice.

Bottom line – AvatarAI is a full‑stack, open‑source platform that covers the entire pipeline from audio capture to animated video output, with both cloud‑API and completely local execution paths. It is the most feature‑complete project in the niche of real‑time AI avatars, ready for production deployment or further research.