joey-zhou/xiaozhi-esp32-server-java

What it solves

This project provides an enterprise-grade Java backend and management platform for ESP32-based smart hardware devices. It bridges the gap between low-power hardware and complex AI capabilities, offering a scalable way to manage devices and handle real-time voice interactions.

How it works

The system uses a dual-process architecture consisting of a management server (xiaozhi-server) for REST APIs and device management, and a dialogue service (xiaozhi-dialogue) for handling real-time audio streams via WebSocket and MQTT. It integrates multiple AI platforms for LLMs, STT (Speech-to-Text), and TTS (Text-to-Speech), allowing for a full voice-to-voice pipeline.

Who it’s for

• Users who own ESP32 hardware and need a robust management platform.
• Developers looking for an enterprise-level scalable backend for AI-powered IoT devices.
• Individuals wanting to quickly deploy a voice-assistant server.

Highlights

• Multi-AI Integration: Supports OpenAI, Zhipu, Xunfei, Ollama, Dify, and Coze.
• Full Voice Pipeline: Local and cloud-based STT/TTS with voice cloning and real-time interruption support.
• IoT Control: Enables voice-commanded device control and multi-device coordination via Function Calling.
• RAG Knowledge Base: Includes document upload and intelligent retrieval for long-term memory.
• Scalable Architecture: Independent management and dialogue processes allow for separate horizontal scaling.
• Comprehensive Monitoring: Visualizes token usage, latency, and device activity.