78/xiaozhi-esp32

What it solves

XiaoZhi is an open-source framework for building AI-powered voice chatbots on ESP32 hardware. It bridges the gap between low-power embedded devices and powerful large language models (LLMs) like Qwen and DeepSeek, enabling real-time voice interaction and hardware control.

How it works

The project provides firmware for various ESP32 chip platforms (S3, C3, C5, C6, P4) that connects to a backend server via WebSocket or a hybrid MQTT+UDP protocol. It uses ESP-SR for offline wake-word detection and Opus audio streaming for high-quality voice interaction. The system integrates the Model Context Protocol (MCP) to allow the AI to control local device hardware (LEDs, servos, GPIO) and access cloud-side capabilities like smart home control or knowledge search.

Who it’s for

• Hardware enthusiasts and DIYers who want to build their own AI voice assistants or robot dogs.
• Developers looking to integrate LLMs into embedded systems using the ESP32 ecosystem.
• Product designers prototyping AI-driven IoT devices.

Highlights

• Broad Hardware Support: Compatible with over 170 release variants and 138 board directories, including M5Stack and Espressif ESP-BOX.
• MCP Integration: Uses the Model Context Protocol for both device-side and cloud-side tool use and control.
• Multimodal Input: Supports camera vision input and speaker recognition to identify different users.
• Flexible Networking: Supports Wi-Fi, wired Ethernet, USB RNDIS, and 4G networking.
• Real-time Interaction: Supports full-duplex interaction on AEC-capable hardware using real-time end-to-end voice models.
• Customizable: Includes a web-based generator for custom wake words, fonts, and emojis.