xinnan-tech/xiaozhi-esp32-server

What it solves

This project provides a comprehensive backend server for the xiaozhi-esp32 open-source intelligent hardware project. It enables ESP32-based devices to function as AI assistants by handling the complex processing of voice interaction, visual perception, and device control that cannot be performed on the hardware itself.

How it works

The server acts as a gateway between the ESP32 hardware and various AI services. It uses MQTT, UDP, and WebSocket protocols to communicate with devices. The system integrates several modular components:

• Voice Pipeline: Combines Automatic Speech Recognition (ASR), Voice Activity Detection (VAD), and Text-to-Speech (TTS) to enable real-time audio interaction.
• Intelligence Core: Connects to Large Language Models (LLMs) and Vision-Language Models (VLLMs) for reasoning and multimodal interaction.
• Extended Capabilities: Implements a memory system (short-term and intelligent retrieval), a RAG-based knowledge base (via RAGFlow), and an intent recognition system for function calling.
• Management: Includes a Web-based control panel for user, device, and system configuration.

Who it’s for

• Developers and hobbyists who own ESP32 hardware and want to host their own private AI assistant backend.
• Users looking to build custom smart home integrations using the MCP (Model Context Protocol) or IoT protocols.

Highlights

• Multimodal Interaction: Supports both voice and visual perception (VLLM).
• Voiceprint Recognition: Can identify different speakers to provide personalized responses.
• Flexible Integration: Supports a wide array of third-party APIs (Aliyun, DeepSeek, OpenAI, etc.) and local deployments (Ollama, FunASR).
• Low Latency: Offers a "streaming configuration" that significantly reduces response times by approximately 2.5 seconds.
• Extensible: Features a plugin system for adding custom tools and function calling capabilities.