ardha27/AI-Waifu-Vtuber

AI Vtuber for Streaming on Youtube/Twitch

What it solves

This project creates an interactive AI virtual YouTuber (VTuber) that can engage in real-time conversations with a user or a Twitch audience. It automates the pipeline of hearing a voice, generating a character-consistent response, and speaking that response back through a virtual avatar.

How it works

The system integrates several AI services to create a seamless loop:

  1. Speech-to-Text: Uses OpenAI Whisper to transcribe spoken audio into text.
  2. Brain: Uses OpenAI's LLM to generate responses based on a customizable identity/lore file.
  3. Translation: Optionally uses DeepL or Google Translate to convert responses into the target language (e.g., Japanese).
  4. Text-to-Speech: Uses VoiceVox (for Japanese) or Silero TTS (for multiple languages including English, Russian, and Spanish) to generate the voice.
  5. Visuals: Connects to VtubeStudio via a virtual audio cable to animate a character avatar based on the audio output.
  6. Twitch Integration: Can read and respond to messages from a Twitch chat stream.

Who it’s for

Content creators and streamers who want an AI-driven virtual persona to interact with their viewers or act as a voice-controlled assistant.

Highlights

  • Multi-language support: Compatible with various TTS engines for global language options.
  • Character Customization: Allows users to define the AI's personality and lore via text files.
  • Twitch Chat Integration: Capable of reading and answering viewers during live streams.
  • Avatar Sync: Designed to work with VtubeStudio for visual representation.

Related

  • Project
    Open-LLM-VTuber/Open-LLM-VTuber

    An open-source, voice-interactive AI companion featuring a Live2D avatar and visual perception, capable of running entirely offline for private, real-time conversations.

  • Project
    lipku/LiveTalking

    A real-time interactive streaming engine for digital humans that synchronizes audio and video for lifelike AI-driven dialogue.

  • Project
    tegnike/aituber-kit

    An all-in-one toolkit for building web applications featuring interactive AI characters with support for various LLMs, character models, and voice synthesis engines.

  • Project
    VRCWizard/TTS-Voice-Wizard

    An accessibility tool for VRChat and other environments that converts speech-to-text and text-to-speech, providing real-time translation and avatar integration.

  • Project
    morettt/my-neuro

    A workbench for creating personalized, human-like AI companions with customizable voices, personalities, and Live2D avatars, supporting local LLM inference and gaming integration.