78/xiaozhi-esp32

An MCP-based chatbot | 一个基于MCP的聊天机器人

What it solves

XiaoZhi is an open-source AI chatbot framework for ESP32 microcontrollers, designed to bring large language models (LLMs) like Qwen and DeepSeek into physical hardware. It solves the challenge of creating a voice-interactive AI assistant that can interact with the physical world through sensors, displays, and IoT control.

How it works

The system uses an ESP32-series chip as the client, which handles audio capture and playback. It connects to a backend server (official or self-hosted) via WebSocket or a hybrid MQTT+UDP protocol. The pipeline consists of an offline voice wake-up (using ESP-SR), followed by a streaming ASR (Automatic Speech Recognition), LLM processing, and TTS (Text-to-Speech) pipeline. It also implements the Model Context Protocol (MCP) to allow the AI to control hardware (like LEDs, servos, and GPIOs) and access cloud-side capabilities (like smart home control or knowledge search).

Who it’s for

Hardware enthusiasts, DIYers, and developers who want to build their own AI-powered voice assistants or robotic devices using ESP32 boards (such as ESP32-S3, P4, or M5Stack CoreS3).

Highlights

  • Broad Hardware Support: Supports over 130 board directories and 170 release variants across various ESP32 chip platforms.
  • Multi-modal Interaction: Integrates camera vision input, OLED/LCD displays with emoji support, and speaker recognition.
  • Flexible Networking: Supports Wi-Fi, wired Ethernet, USB RNDIS, and 4G networking.
  • MCP Integration: Uses the Model Context Protocol for both device-side hardware control and cloud-side capability extension.
  • C-based Firmware: Built on ESP-IDF v6.0.1+, ensuring high performance on embedded systems.

Related

  • Project
    xinnan-tech/xiaozhi-esp32-server

    A backend server for the xiaozhi-esp32 project that enables ESP32 devices to become AI assistants with voice, vision, and smart home control capabilities.

  • Project
    joey-zhou/xiaozhi-esp32-server-java

    A Java-based enterprise backend for ESP32 smart hardware, providing a management platform and AI dialogue services with RAG, voiceprint recognition, and multi-LLM support.

  • Project
    shenjingnan/xiaozhi-client

    A client that connects AI servers to Model Context Protocol (MCP) servers, enabling AI agents to access external tools and data sources.

  • Project
    Lucassssss/eechat

    A secure local AI chat application that supports the Model Context Protocol (MCP) for easy integration of AI tools and services while keeping data stored locally.

  • Project
    akdeb/ElatoAI

    A framework for deploying real-time voice AI on ESP32 devices, connecting low-power hardware to frontier speech-to-speech models via edge functions.