Voine/ChatWaifu_Mobile

What it solves

This project provides an Android application that transforms a standard LLM chat experience into an immersive, character-based interaction. It integrates text generation, voice synthesis, and 2D animation to create a virtual companion with a specific personality.

How it works

The app combines several specialized technologies to create a live interaction loop:

• Text Generation: Uses ChatGPT (via OpenAI API) for the conversational logic and character personas.
• Voice Synthesis: Implements local VITS-ncnn for text-to-speech, allowing the character to speak with specific voices (e.g., from Blue Archive or Steins;Gate).
• Visuals: Uses the Native Live2D SDK to render animated 2D characters on screen.
• Input: Employs local Sherpa-ncnn for speech-to-text recognition, enabling voice-based chatting.
• Translation: Integrates Baidu Translate to convert GPT's output into Japanese, ensuring compatibility with the Japanese-centric VITS voice models.

Who it’s for

Anime fans and Android developers interested in integrating multiple AI modalities (voice, text, and animation) into a single mobile application.

Highlights

• Local Inference: Voice synthesis and speech recognition are handled locally on the device using ncnn libraries.
• Customizable Characters: Users can import their own VITS and Live2D models to create new characters.
• Multimodal Integration: Combines LLM text, local TTS, and local STT in a single mobile UI built with Jetpack Compose.
• Persona Support: Includes built-in character settings and supports external persona configurations.