coderzc/open-xiaoai-bridge

Open‑XiaoAI Bridge – a bridge between Xiaomi XiaoAI speakers and external AI services

What it is – A server‑side component (written in Python 3.12+ with a Rust client) that runs on a host (Docker or bare‑metal) and talks to a patched XiaoAI speaker over a WebSocket. It intercepts the speaker’s microphone/audio stream, performs wake‑word detection, VAD and optional offline ASR, then forwards the recognised text to one of several configurable AI back‑ends. The reply is turned into speech (via the speaker’s native TTS, Doubao cloud TTS, OpenAI‑compatible TTS or a local MLX‑Audio model) and sent back to the speaker for playback.

Core capabilities (as listed in the README)

Feature	What it does
OpenAI‑compatible services	Connects to any `/v1/chat/completions` endpoint (Hermes Agent API Server, OpenAI, Ollama, LM Studio, etc.).
QwenPaw integration	Sends recognised text to an Alibaba QwenPaw agent via its HTTP console API and plays the agent’s reply.
OpenClaw integration	Routes commands to an OpenClaw gateway, supporting both single‑shot and continuous multi‑turn conversations.
XiaoZhi AI integration	Uses the `xiaozhi‑esp32‑server` protocol for real‑time audio streaming.
Custom wake‑words	Multiple Chinese/English wake‑words can be defined; each can be mapped to a different AI service or OpenClaw agent.
Multi‑agent routing	One speaker, many wake‑words → each wake‑word selects a distinct OpenClaw session, enabling “zero‑overhead” switching.
Continuous dialogue	After the wake‑word, the bridge keeps the conversation alive without needing to repeat the wake‑word; saying “小爱同学” interrupts.
VAD + KWS front‑end	Voice‑activity detection reduces spurious recognitions and saves power.
HTTP API	Remote control of the speaker – play text, URLs, files, trigger TTS, interrupt playback, query status, health check.
Modular design	Every connector (XiaoZhi, OpenClaw, OpenAI, QwenPaw, API server, TTS) can be turned on/off via environment variables.

Quick‑start (from the README)

Prepare the speaker – flash a newer firmware that enables SSH and install the Rust client (packages/client‑rust).
Download models – VAD + KWS (always) and optionally Sherpa ASR if you want local offline transcription. Model zip is provided in the releases page.

Run with Docker Compose (recommended)

curl -O https://raw.githubusercontent.com/coderzc/open-xiaoai-bridge/main/config.py
curl -O https://raw.githubusercontent.com/coderzc/open-xiaoai-bridge/main/docker-compose.yml
# edit config.py / docker‑compose.yml as needed
docker compose up -d

Mounts ./models into the container and exposes the API on port 9092.

Or run locally – after cloning the repo, install uv (Python) and Rust, place the model files under core/models/, then start:
```
API_SERVER_ENABLE=1 XIAOZHI_ENABLE=1 OPENCLAW_ENABLE=1 OPENAI_ENABLE=1 QWENPAW_ENABLE=1 ./scripts/start.sh
```
Optional token‑based client authentication can be enabled with OPEN_XIAOAI_TOKEN.
Configure services – edit config.py. Each backend has a block (openai, openclaw, qwenpaw, xiaozhi) where you set the endpoint URL, API key, model name, TTS provider, etc.

Environment variables (summary)

Variable	Effect	Default
`XIAOZHI_ENABLE`	Enable XiaoZhi AI backend	disabled
`OPENCLAW_ENABLE`	Enable OpenClaw connector	disabled
`OPENAI_ENABLE`	Enable generic OpenAI‑compatible connector	disabled
`QWENPAW_ENABLE`	Enable QwenPaw connector	disabled
`API_SERVER_ENABLE`	Turn on the HTTP control API	disabled
`AUDIO_INPUT_ENABLE`	Turn on microphone processing (required for VAD/KWS/ASR)	enabled
`API_SERVER_HOST` / `API_SERVER_PORT`	Host/port for the HTTP API	`127.0.0.1` / `9092`
`OPEN_XIAOAI_TOKEN`	Shared secret for client‑side authentication	none (no auth)
`CONFIG_PATH`	Path to custom `config.py`	`./config.py`
`LOGLEVEL`	Logging verbosity	`INFO`

API Server (when `API_SERVER_ENABLE=1`)

Method	Path	Purpose
`POST`	`/api/play/text`	Synthesize text with the selected TTS and play it on the speaker
`POST`	`/api/play/url`	Stream an audio file from a public URL
`POST`	`/api/play/file`	Upload a local audio file and play it
`POST`	`/api/tts/doubao`	Direct Doubao TTS call (cloud voice)
`GET`	`/api/tts/doubao_voices`	List available Doubao voice IDs
`POST`	`/api/wakeup`	Programmatically trigger the wake‑up flow
`POST`	`/api/interrupt`	Stop current playback/TTS
`GET`	`/api/status`	Current playback state
`GET`	`/api/health`	Simple health check

Example usage (curl):

curl -X POST http://localhost:9092/api/play/text \
  -H "Content-Type: application/json" \
  -d '{"text": "你好，我是小爱同学"}'

TTS provider hierarchy

Each backend (openai, openclaw, qwenpaw) can specify a tts_provider:
- xiaoai – use the speaker’s built‑in TTS (no extra config)
- doubao – Volcano Engine cloud TTS (requires app_id/access_key)
- openai – any OpenAI‑compatible /v1/audio/speech service
- mlx_audio – local MLX‑Audio server (useful on Apple Silicon)
If tts_provider is omitted, the original logic applies: XiaoAI uses native TTS, others fall back to Doubao.

How the bridge works (high‑level flow)

Audio capture – The Rust client on the speaker streams PCM audio over a WebSocket (open_xiaoai_server:4399).
Pre‑processing – The server runs VAD and KWS on the incoming stream. When a wake‑word matches, the WakeupSessionManager starts a session.
ASR – Depending on input_mode you get either:
- local_asr – VAD → Sherpa ASR (offline) → text
- xiaoai_asr – the speaker’s native ASR is handed over to the bridge.
Routing – before_wakeup hook (user‑editable in config.py) can return a string such as "openai", "openclaw", "qwenpaw" to select the target backend for this session.
Backend interaction – The selected connector sends the text to its remote service, receives a response, optionally runs TTS, and streams the audio back to the speaker via the same WebSocket.
Continuous dialogue – The XiaoAIConversationController monitors ASR/AudioPlayer events; saying “小爱同学” or a configured exit phrase ends the session.

Extending / customizing

Custom wake‑words – Define new keywords in the KWS configuration; each can map to a different backend by returning the appropriate identifier in before_wakeup.
Multi‑agent routing – By assigning different wake‑words to different OpenClaw agents you can switch agents on‑the‑fly without restarting the bridge.
Docker networking – If the bridge container needs to reach services running on the host, either use the host.docker.internal alias (Mac) or set network_mode: host.
Authentication – Set OPEN_XIAOAI_TOKEN to require the same token on the Rust client side; otherwise any client can connect.

Who might use this?

XiaoAI hobbyists who want their smart speaker to talk to ChatGPT, Claude, LLaMA, or custom agents.
Researchers building voice‑first agents who need a cheap, always‑on hardware front‑end.
Home‑automation enthusiasts who want to trigger local services (e.g., Home Assistant) via voice without relying on Xiaomi’s cloud.

License & community

License: MIT (see LICENSE).
Stars: shown on the badge; the project is actively maintained.
Docker image: published on GitHub Container Registry (ghcr.io/coderzc/open-xiaoai-bridge).
Demo videos: two Bilibili links are provided in the README.

Bottom line

Open‑XiaoAI Bridge is a genuine, open‑source integration layer that lets a Xiaomi XiaoAI speaker act as a front‑end for a wide range of modern LLM/agent services. It handles wake‑word detection, optional offline speech‑to‑text, routing to multiple back‑ends, and flexible TTS output, all configurable via a simple config.py and environment variables.