uezo/aiavatarkit

What it solves

AIAvatarKit provides a high-performance framework for building conversational AI avatars that can interact across multiple channels (web, mobile, phone, and hardware devices) with ultra-low latency. It eliminates the complexity of stitching together separate voice activity detection (VAD), speech-to-text (STT), large language models (LLMs), and text-to-speech (TTS) services into a cohesive, real-time streaming pipeline.

How it works

The project uses a modular STSPipeline (Speech-to-Speech) that streams data through four main stages: VAD $\rightarrow$ STT $\rightarrow$ LLM $\rightarrow$ TTS.

• Adapters: These wrap the pipeline for specific transport layers (e.g., WebSockets, HTTP, Twilio, LINE), allowing a single conversation to persist even if a user switches channels.
• Modular Components: Every stage is swappable. Users can plug in different providers for each step (e.g., using Azure for STT and Claude for the LLM).
• Agentic Capabilities: It supports tool calling and the Model Context Protocol (MCP), enabling avatars to execute background tasks or fetch real-time data without stalling the conversation.
• Expressiveness: Beyond audio, the pipeline can trigger facial expressions, animations, and on-screen visual artifacts (like maps or charts) simultaneously with the spoken response.

Who it’s for

Developers building interactive AI characters for web/mobile apps, virtual store staff for digital signage, companion hardware (like Raspberry Pi), metaverse characters (VRChat), or automated phone operators.

Highlights

• Ultra-low latency: Achieves under 1 second from the end of user speech to the first audio response.
• Omnichannel support: One pipeline serves web, phone, and messaging apps, maintaining conversation state across them.
• Production-ready tooling: Includes an Admin Panel for configuration, logs, and metrics, and integrates with Langfuse for tracing.
• Extensive provider support: Built-in integrations for a wide array of LLMs (GPT, Claude, Gemini, Grok), TTS engines (VOICEVOX, ElevenLabs, Azure), and STT services.