garycaogan/ai-live-avatar

What it is

ai-live-avatar is a real, runnable project that lets an AI-generated digital human host its own live stream. Viewers send chat messages (danmaku), and the AI avatar replies, changes outfits, or performs gestures in real time — all rendered by the Vidu S2 real-time digital human service. This repo is the "middle layer" that makes that possible: it filters and schedules incoming comments, manages the avatar's persona and wardrobe, renders the stream overlay, provides a local control workbench, and outputs to OBS or direct streaming.

How it works

• Vidu S2 generates the avatar's visuals, voice, and spoken replies. This project doesn't train or render the avatar itself — it orchestrates Vidu's API.
• Comment pipeline: incoming danmaku passes through a filter (blocklist in moderation.json), then a rule-based router (live/router.py) classifies each message as chat, outfit change, prop, background, action, or restore. The router uses no LLM — it's fast and free.
• Queueing: comments are queued so the avatar finishes one reply before handling the next. Gifts don't jump the queue.
• Wardrobe: each outfit is a complete photo (same room, lighting, composition as the base image). Changing outfits restarts the Vidu session with the new photo, taking ~6 seconds.
• Actions: keywords trigger small gestures (heart, wave, blink). Idle small talk comes from a persona-defined topic pool.
• Overlay & streaming: a stage page provides the avatar feed, a transparent overlay layer, and info cards. Output goes to OBS (via browser source or WebSocket control) or direct RTMP streaming.

What you need

• macOS/Linux, Python 3.11+, ffmpeg with libx264 and aac
• A Vidu API key (paid; roughly 169 RMB/hour at official rates)
• Optional: Evolink API key for generating character images/outfits/backgrounds and voice cloning; OBS 30+; a YouTube stream key
• Your own character assets — the author's images, voices, and personas are not included (gitignored)

Getting started

1. Clone, create a venv, install requirements, fetch the Alibaba Cloud RTC vendor SDK, copy config.example.json to config.json.
2. Store API keys in ~/.deepsea/ with restricted permissions.
3. Run python server.py; open http://127.0.0.1:28900/workbench for the workbench.
4. Create a character: upload a base image, write a persona and opening line, add outfits, pick/clone a voice, configure action keywords.
5. Connect OBS (browser source URLs provided) or stream directly. A mock mode (tests/dev_mock_stack.py) lets you test the full flow without spending Vidu credits.

Notable design decisions

• Background/prop changes are disabled by default: in testing, mid-stream visual edits caused the avatar's clothes to distort. They can be enabled via config.json but the author warns against using them in public streams.
• Auto-recovery: if Vidu drops mid-stream, the server can restart the session after 3 seconds (max 6/hour), unless the drop was due to insufficient credits or content moderation.
• Danmaku sources: YouTube chat via a provided bridge script; Chinese platforms require their official live-interaction APIs (not web scraping). The project doesn't include Douyin integration.
• Compliance: the overlay permanently shows an "AI real-time generated" label, required by platform rules for AI avatar streams.

Bottom line

This is a functional, MIT-licensed tool for running an AI-hosted live stream with real-time chat interaction, outfit changes, and gestures — not a demo or tutorial. It's practical but requires paid API credits, your own character assets, and some setup effort.