morettt/my-neuro

This project lets you create your own AI desktop companion with customizable characters and voice conversations that respond in just 1 second. Features include long-term memory, visual recognition, voice cloning and LLM training. Compatible with various Live2D customizations.

What is my‑neuro?

my‑neuro is an open‑source “AI companion” platform that lets you create a personal virtual character that can talk, show facial expressions, and even interact with games. The idea is to let users feed their own data (voice recordings, personality prompts, etc.) and assemble a character that behaves like a human‑like friend – speaking with a chosen voice, remembering past conversations, showing emotions, and reacting to visual input.

Core capabilities (as listed in the README)

Local, low‑latency inference – the whole stack runs on your own machine, aiming for sub‑second response times.
Speech‑to‑text & text‑to‑speech – subtitles are shown in Chinese while the audio can be in any language. The default TTS backend is the open‑source GPT‑SoVITS project.
Voice & persona customization – you can switch between male/female voices, load different character “voice‑fonts”, and define personality traits.
Real‑time interruption – both voice and keyboard input can cut the AI off while it is speaking.
Mouth/face animation – the character displays different facial expressions and body motions based on the dialogue.
Vision integration – the system can run image‑recognition models and decide, from the language intent, when to activate visual processing.
Desktop control – voice commands can launch applications on the host computer.
AI‑generated singing – a sponsored feature that lets the avatar sing.
Game interaction – experimental support for playing Minecraft, visual‑novel style games (galgame) and other “draw‑and‑guess” style games, using the mindcraft bot as a bridge.
Memory system – a long‑term memory module (based on MemOS) stores key facts about the user so the AI can recall preferences, moods, etc.
Planned extensions (still TODO):
- Emotion engine that simulates a persistent mood state.
- Live streaming to platforms like Bilibili.
- Web UI, Android app, and broader live‑2D avatar support.
- Online search for up‑to‑date information.
- More sophisticated dialogue initiation (AI starts conversations on its own).

How it’s built

Component	Technology / Source
LLM backend	Supports both open‑source fine‑tuned models (run locally) and closed‑source APIs (e.g., OpenAI, Claude).
TTS	GPT‑SoVITS (open‑source Chinese/Multilingual voice synthesis).
Vision	Plug‑in for any image‑recognition model; the README does not specify a particular framework, but the architecture expects a model that can return textual intents.
Memory	MemOS from the MemTensor project – a vector‑store‑style long‑term memory.
Desktop automation	Playwright‑MCP – a Microsoft Playwright‑based tool for controlling the UI via a web‑socket protocol.
Game bots	mindcraft – a Minecraft‑automation bot that can be driven by the AI’s output.
Frontend	Currently a desktop client with subtitles; a web UI is “already done, will be integrated soon”.
Operating system	Designed for Windows/Linux desktop environments; the README mentions an Android “肥牛” app as a future target.

Project status

The README claims ~30 % of the roadmap is already functional (personality definition, memory, basic voice output, MCP integration, some game bots).
Several items are marked as completed (low‑latency local inference, subtitle sync, voice switching, MCP support, real‑time interruption, expression rendering, vision hooks, TTS integration, desktop control, AI singing).
Many ambitious features are still TODO (persistent emotion model, streaming, web UI, Android app, full long‑term memory, autonomous dialogue, etc.).
The roadmap includes a target of ~60 % completion by 1 June and a goal to finish all listed ideas within the year.
The project has a modest community (contributors badge, a Discord‑style QQ group, and a few monetary sponsors).

Who might use it?

Enthusiasts who want a highly customizable AI avatar that runs locally and can be tailored with their own voice recordings.
Content creators looking for a virtual “partner” that can speak, sing, and react in live streams.
Developers interested in experimenting with multimodal AI (speech, vision, memory) and integrating it with desktop automation or game bots.
Researchers exploring personal‑assistant interaction, emotion modeling, or long‑term memory in LLM‑driven agents.

Getting started (as per the README)

Visit the official tutorial site – http://mynewbot.com/tutorials for deployment instructions.
Clone the repo and follow the setup scripts (not detailed in the README, but typical steps would include installing Python dependencies, downloading a base LLM, and configuring the GPT‑SoVITS TTS model).
Choose a pre‑packaged character (the “肥牛” avatar) or train your own voice/personality using the provided tools.
Run the desktop client; you can interact via microphone or keyboard, see subtitles, and watch the avatar’s expressions.

Quick take‑away

my‑neuro is a genuine, ambitious open‑source effort to build a locally‑run, multimodal AI companion that can talk, show emotions, remember you, and even play games. It stitches together several existing open‑source pieces (GPT‑SoVITS for voice, MemOS for memory, mindcraft for Minecraft, Playwright‑MCP for UI control) and adds glue code to let a user define a custom persona. While many of the “human‑like” features are still in the roadmap, the core pipeline (LLM → text → TTS + subtitles + avatar animation) is already functional. If you’re comfortable tinkering with AI models and want a personalizable virtual friend that runs on your own hardware, this project is worth exploring.