uezo/ChatdollKit

ChatdollKit enables you to make your 3D model into a chatbot

What it solves

ChatdollKit is a 3D virtual assistant SDK that allows developers to transform 3D models (specifically VRM models) into voice-enabled, interactive chatbots. It bridges the gap between Large Language Models (LLMs) and 3D avatars by synchronizing AI-generated text with speech, lip-syncing, facial expressions, and body animations in real-time.

How it works

The SDK integrates several AI components into a Unity-based workflow:

  • LLM Integration: Connects to services like ChatGPT, Claude, Gemini, and Dify to generate conversational responses. It supports function calling and multimodal capabilities.
  • Speech Pipeline: Uses Speech-to-Text (STT) for listening (e.g., OpenAI, Azure, Google) and Text-to-Speech (TTS) for speaking (e.g., VOICEVOX, Style-Bert-VITS2). It includes Voice Activity Detection (VAD) to handle turn-taking and noise resistance.
  • Avatar Control: A ModelController manages the 3D model's state, triggering facial expressions and animations based on tags (e.g., [face:Joy]) embedded in the LLM's response.
  • Synchronization: It synchronizes the audio output with lip-syncing and autonomous blinking to create a lifelike presence.

Who it’s for

  • Unity Developers: Those looking to build virtual agents, AI Vtubers, or interactive NPCs.
  • App Creators: Developers building VR, AR, or WebGL applications that require a conversational 3D interface.
  • AI Researchers/Hobbyists: People wanting to embody an LLM in a visual, emotive 3D character.

Highlights

  • Multi-Platform Support: Compatible with Windows, Mac, Linux, iOS, Android, VR, AR, and WebGL.
  • Natural Interaction: Supports "barge-in" (interrupting the AI mid-sentence) and WebSocket streaming STT to reduce latency.
  • Emotive Control: Autonomous switching of voice styles, facial expressions, and gestures based on conversation context.
  • Memory & State: Supports long-term memory for conversation history and dynamic multi-language switching.
  • Agentic Capabilities: Integration with Dify and AIAvatarKit allows the avatar to act as an AI agent with tool-calling abilities.

Related

  • Project
    datawhalechina/torch-rechub

    Torch-RecHub is a PyTorch framework for building recommender systems with 30+ models, modular design, ONNX export, and support for multiple hardware backends.

  • Project
    GangTailorUpgrade/undress-service

    A self-hosted AI fashion platform that digitizes your wardrobe and uses generative AI to recommend and visualize outfits based on weather, occasion, and style.

  • Project
    OpenTSLM/OpenTSLM

    OpenTSLM is an open‑source library that adds native time‑series handling to Llama 3.2 and Gemma LLMs, enabling natural‑language prompting and reasoning over medical signals (ECG, EEG, accelerometer, etc.). It provides pre‑trained checkpoints, a simple `OpenTSLM` Python API, demo scripts for inference on several benchmark tasks, and a curriculum‑learning training pipeline covering MCQ, captioning, and chain‑of‑thought reasoning stages. Install via `pip install opentslm`, load models from Hugging‑Face, and fine‑tune or evaluate using the supplied scripts. Licensed under MIT.

  • Project
    ZJU-REAL/Easel

    Easel is an open‑source AI‑driven content workbench for social‑media creators. It combines an OpenClaw LLM agent, per‑account profiles, and 113+ executable “skills” to discover trends, plan topics, generate text, images, audio and video, publish directly to seven Chinese platforms, and feed performance data back into the profile for continual improvement.