RachelForster/Shinsekai

Shinsekai – AI‑powered desktop assistant for visual‑novel style RPGs

What it is – Shinsekai (新世界) is a locally‑run desktop application that lets you chat with characters from galgames, otome games, or story‑driven RPGs. The dialogue is generated by a large language model (LLM), and the system automatically synchronises the character’s portrait (立绘) and emotional expression, optionally adding voice‑over (TTS) and speech‑to‑text (ASR). All components – LLM, TTS, ASR, image generation, and custom tools – are configurable through a React‑based settings UI and can be extended with plugins.

Core capabilities

Module	What you get
Role & template management	Import/export character packs (`.char`), generate role‑play settings with AI assistance, apply multi‑character chat templates, keep conversation history and save/load sessions.
Portrait & performance	Attach several portrait images per character, tag each with an emotion, and have the UI automatically switch images based on emotion commands in the dialogue.
Voice	Text‑to‑speech via GPT‑SoVITS, Genie TTS, CosyVoice, etc.; optional playback of pre‑recorded voice files. Speech‑to‑text via Vosk by default, with Whisper‑style back‑ends possible through plugins.
LLM & tools	Connect to any OpenAI‑compatible endpoint (DeepSeek, OpenAI, etc.) via the API Settings file. Supports streaming responses and tool‑calling; tools can be supplied by built‑in adapters, plugins, or an external MCP server.
Text‑to‑image (T2I)	Hook a ComfyUI server (or other adapters) to generate character art on the fly; configuration lives in the same API settings pane.
React Settings Center	A web‑based UI (`webui_react.py`) that manages API keys, model IDs, portrait resources, themes, plugins, and MCP connections.
Plugin system	SDK lets developers add new adapters (LLM, TTS, ASR, T2I), new tools, or UI panels. Plugins are listed in `data/config/plugins.yaml` and can be installed/enabled from the Settings UI.
MCP (Model Context Protocol)	Register remote tool services (SSE or stdio) without writing a plugin; the remote tools become part of the current LLM’s tool set.
Theme editor	Visual editor for chat window skins – fonts, colours, layout, name tags, etc.
Local‑first data	All configuration, character packs, histories, and generated assets live under `data/`, making backup and offline use straightforward.

Quick start (≈5 min)

Get the code – git clone https://github.com/RachelForster/Shinsekai && cd Shinsekai (or download a release package).
Install – run the platform‑specific install.* script, or create the Conda environment from environment.yml and pip install -r requirements.txt.
Launch the settings UI – double‑click start-react.* (Windows/macOS/Linux). The script starts a local Python HTTP bridge and opens the React settings page in your browser.
Configure an LLM – in API Settings choose a provider, fill in model ID, API key and base URL, then save.
Add a character – import a .char pack (e.g., nanami.char from the releases) via the Role Management tab.
Create a chat template – select the characters you want, generate a template, and open the main chat window.
Talk – type or speak; the LLM replies, the portrait switches according to emotion tags, and optional TTS reads the line aloud.

Extending Shinsekai

Plugins – place a plugin folder under plugins/ and list it in data/config/plugins.yaml, or install from the built‑in plugin registry UI. Use the provided SDK (python -m sdk.cli create …) to scaffold new adapters, tools, or UI panels.
MCP services – install the mcp Python package, run a compatible server, and point data/config/mcp.yaml (or the Settings UI) at it. The remote tools appear in the LLM’s tool list automatically.
Custom T2I pipelines – configure a ComfyUI endpoint (or any other T2I service) in the API settings, then reference the workflow/node IDs in a character’s portrait generation step.

Build the desktop app (Tauri)

If you want the native desktop wrapper:

cd frontend
pnpm install
pnpm tauri:dev   # for development
pnpm tauri build  # for a release package

The build process bundles the React front‑end, signs the updater (set TAURI_SIGNING_PRIVATE_KEY), and produces installers for the target OS.

License & community

License – source code is open, but redistribution of the compiled app, releases, or commercial use requires written permission (see LICENSE).
Community – QQ group 928985460, issue tracker on GitHub, and a separate plugin‑registry repository.

Bottom line

Shinsekai is a fairly complete, locally‑hosted platform that combines LLM‑driven dialogue, dynamic character portraits, voice synthesis, and extensible tooling for interactive storytelling. It is geared toward hobbyists and creators who want a self‑contained “visual‑novel chat” experience without relying on cloud services for the heavy lifting.