huangjunsen0406/py-xiaozhi

Open-source AI assistant ecosystem with MCP integrations, multimodal workflows, IoT support, and cross-platform voice interaction.

py‑xiaozhi – A lightweight, cross‑platform multimodal AI framework

What it is – py-xiaozhi is a Python library that lets you build an AI‑powered assistant that can listen, speak, see, and control hardware in real time. It is built on asyncio for low‑latency streaming and runs on desktops (Windows/macOS/Linux) as well as on edge devices such as Raspberry Pi, Jetson Nano or Horizon Robotics boards.

Why it matters – The project bridges large‑language‑model (LLM) services with on‑device perception (wake‑word detection, camera capture) and actuation (GPIO, MQTT). In other words, it provides a ready‑made “brain‑body” stack for embodied AI, voice assistants, or robotics prototypes without having to stitch together many separate tools.

Core capabilities

Feature What you get
Real‑time voice AI Opus‑encoded audio streaming, sub‑20 ms latency, async handling of microphone and speaker.
Offline wake‑word Sherpa‑ONNX keyword spotting runs locally, so the assistant can be activated without internet.
Vision‑language Camera capture integrated with a vision‑language model (image understanding / scene perception).
MCP tool ecosystem JSON‑RPC 2.0 “tool” server exposing utilities like music playback, screenshot, weather, volume control, etc.
Cross‑platform UI PySide6 + QML graphical UI, a pure CLI mode, and a GPIO‑only mode for headless embedded boards.
Secure communication WebSocket or MQTT with TLS/WSS, automatic reconnection and device fingerprint authentication.
Plugin architecture Event‑driven async core, dependency‑injection container, easy to add new tools, protocols or UI plugins.
IoT / robotics ready Direct GPIO access, MQTT bridge, and a modular design for sensor/actuator integration.

Typical use‑cases

  • Desktop voice assistant – Run on a laptop or PC with a GUI that shows a floating avatar, handles voice commands, plays music, shows weather, etc.
  • Edge robot controller – Deploy on a Raspberry Pi or Jetson Nano, use the wake‑word to start listening, process camera frames, and drive motors via GPIO.
  • Smart home hub – Connect over MQTT to other devices, expose tool APIs (e.g., turn lights on/off) while the LLM handles natural‑language intent parsing.
  • Research prototype – Quickly prototype multimodal pipelines (speech → LLM → vision → actuation) without building the plumbing from scratch.

Getting started (quick‑start guide)

# 1. Clone the repo
git clone https://github.com/huangjunsen0406/py-xiaozhi.git
cd py-xiaozhi

# 2. Install dependencies (recommended: uv, otherwise pip)
uv sync                # base install (CLI / GPIO mode)
# uv sync --extra gui   # include PySide6 for the graphical UI
# pip install -e .      # alternative for pip users

# 3. Run the assistant
# GUI mode (default when the extra is installed)
python main.py

# CLI‑only mode (no GUI, useful on headless boards)
python main.py --mode cli

# Choose communication protocol (default is WebSocket)
python main.py --protocol mqtt

Documentation – Full startup tutorials, configuration reference, and API docs are hosted at https://huangjunsen0406.github.io/py-xiaozhi/. A video walkthrough is also available on Bilibili.

Architecture at a glance

  • Event‑driven async core (asyncio loop) – all I/O (audio, network, camera) runs without blocking.
  • Layered design – application logic → protocol layer (WebSocket/MQTT) → UI layer (PySide6/CLI/GPIO).
  • Dependency injection – a bootstrap container creates and wires components, making plugins easy to add.
  • Security – TLS‑encrypted channels, device fingerprinting, and per‑tool permission checks.

Extending the framework

  1. Add a new MCP tool – drop a Python module under src/mcp/tools/ implementing the required JSON‑RPC methods.
  2. Support a new protocol – subclass the abstract Protocol class in src/protocols/ and register it.
  3. Create a plugin – place code in src/plugins/ and declare it in the plugin manifest; the core will load it automatically.

Community & support

  • Sponsors – GitDo.net, Token能量站, 良心AI (provide aggregated API keys for Claude, Gemini, GPT, etc.).
  • Contributing – see CONTRIBUTING.md for the workflow; the project follows a typical PR‑review‑CI cycle.
  • Demo – a short Bilibili video showcases the UI and voice interaction.

License

py-xiaozhi is released under the permissive MIT License.

Bottom line – If you need a ready‑made, async‑first Python stack that couples LLM chat, speech I/O, vision, and hardware control, py-xiaozhi provides a solid, cross‑platform foundation with both GUI and headless options.

Related

  • Project
    MervinPraison/PraisonAI

    PraisonAI is an open‑source framework for building autonomous, self‑improving AI agents. It implements a five‑layer stack (prompt, context, harness, loop, graph) plus a managed‑agents layer that lets you run agents locally or in remote sandboxes (Docker, e2b, Modal, etc.). The Python SDK (`praisonaiagents`) provides declarative agents, tool integration via the MCP protocol, loop‑control (budgets, doom‑loop detection), and graph orchestration (routing, parallel, repeat). A CLI, visual flow builder, dashboard, and JavaScript SDK round out the ecosystem. Supports 100+ LLM providers and is MIT‑licensed.

  • Project
    xinnan-tech/xiaozhi-esp32-server

    A backend server for the xiaozhi-esp32 project that enables ESP32 devices to become AI assistants with voice, vision, and smart home control capabilities.

  • Project
    78/xiaozhi-esp32

    An open-source ESP32-based AI voice chatbot framework that integrates LLMs and the Model Context Protocol (MCP) for real-time voice interaction and hardware control.

  • Dispatch
    Qwen 3.5‑397B‑A17B release: hybrid linear‑attention MoE model with 1 M token context and state‑of‑the‑art multimodal performance

    Qwen 3.5‑397B‑A17B is a 397 billion‑parameter multimodal model that activates only 17 billion parameters per token, delivering state‑of‑the‑art performance on language, coding, reasoning and vision tasks while being up to 19× faster than its predecessor.

  • Project
    modelscope/ms-agent

    MS‑Agent is a lightweight Python framework for building autonomous LLM agents with tool‑calling, memory, multimodal input, and a skill‑retrieval system. It uses ModelScope’s Model Calling Protocol (MCP) for LLM access, provides a React/FastAPI web UI, and includes ready‑made projects for code generation, short video creation, and deep‑research workflows.