uezo/aiavatarkit

🥰 Building AI-based conversational avatars lightning fast ⚡️💬

What it solves

AIAvatarKit provides a framework for building low-latency, conversational AI avatars that can interact across multiple channels (web, mobile, phone, and hardware devices). It solves the complexity of orchestrating the pipeline between voice activity detection (VAD), speech-to-text (STT), large language models (LLMs), and text-to-speech (TTS) while maintaining a responsive, human-like conversation flow.

How it works

The project uses a modular STSPipeline (Speech-to-Speech) that streams data through four primary stages:

  1. VAD: Detects when a user is speaking and manages turn-taking using semantic gates (e.g., Namo Turn) to determine when a user has actually finished their thought.
  2. STT: Converts audio to text using providers like Azure, Google, or OpenAI.
  3. LLM/Agent: Processes the text and generates a response, supporting tool calls and MCP (Model Context Protocol) for external data access.
  4. TTS: Converts the response back to audio using services like VOICEVOX, ElevenLabs, or Azure.

Adapters wrap this pipeline to enable omnichannel deployment, allowing a single conversation to persist as a user moves between different interfaces (e.g., from a phone call to a LINE bot).

Who it’s for

Developers building interactive AI characters for web/mobile apps, virtual store staff for digital signage, companion hardware (Raspberry Pi), metaverse avatars (VRChat), or AI-powered phone operators.

Highlights

  • Ultra-low latency: Achieves sub-1-second response times from the end of speech to the first audio output.
  • Modular Architecture: All components (VAD, STT, LLM, TTS) are swappable, allowing developers to upgrade models or voices without rewriting the system.
  • Omnichannel Support: A single pipeline can serve WebSockets, HTTP, telephony (Twilio/Asterisk), and messaging apps (LINE).
  • Multimodal Expressiveness: Supports facial expressions, animations, and on-screen artifacts (like charts or maps) triggered by the LLM.
  • Production Ready: Includes an Admin Panel for real-time configuration, logs, metrics, and guardrails that can interrupt the avatar mid-sentence to correct errors.

Related

  • Project
    huggingface/speech-to-speech

    speech‑to‑speech is an open‑source Python toolkit that runs a full voice‑assistant locally. It chains VAD → speech‑to‑text → LLM → text‑to‑speech, with many interchangeable model back‑ends, and speaks the OpenAI Realtime protocol so existing clients can connect to your own machine.

  • Project
    PunithVT/ai-avatar-system

    AvatarAI is an open‑source, production‑grade system that turns any portrait photo and a short voice sample into a real‑time, lip‑synced talking avatar. It streams Whisper‑transcribed speech to a selectable LLM (Claude, GPT‑4o, or local Ollama/vLLM), feeds the generated text to a multilingual TTS (Chatterbox/XTTS), and renders video frames with MuseTalk V1.5. The stack includes Next.js front‑end, FastAPI back‑end, PostgreSQL, Redis, Celery, Prometheus, and Docker‑Compose, with optional one‑click AWS GPU deployment. Features include zero‑shot voice cloning, barge‑in interruption, multi‑LLM support, JWT auth, conversation memory summarisation, and optional face‑restoration via GFPGAN.

  • Project
    datascale-ai/opentalking

    An open-source orchestration framework for real-time digital-human conversations, integrating LLMs, TTS, STT, and video rendering via WebRTC.

  • Project
    dnhkng/GLaDOS

    A character-driven voice assistant that recreates GLaDOS's personality using a multi-core architecture for emotion, memory, vision, and proactive responses.

  • Project
    livekit/agents-js

    A Node.js framework for building real-time, multi-modal voice agents that can see, hear, and understand using WebRTC.