joey-zhou/xiaozhi-esp32-server-java

小智ESP32的Java企业级管理平台，提供设备监控、音色定制、角色切换和对话记录管理的前后端及服务端一体化解决方案

What it solves

Xiaozhi ESP32 Server Java provides a professional, enterprise-grade backend for ESP32-based smart hardware devices. It bridges the gap between simple hardware prototypes and a scalable system capable of managing multiple devices, integrating various AI models, and handling real-time voice interactions.

How it works

The project uses a dual-process architecture to separate concerns:

  • Management Server (xiaozhi-server): Handles administrative tasks, REST APIs, user/device management, and OTA (Over-the-Air) updates.
  • Dialogue Server (xiaozhi-dialogue): Manages the real-time AI pipeline, including WebSocket/MQTT communication for audio streams, speech-to-text (STT), large language models (LLMs), and text-to-speech (TTS).

It integrates with a wide array of AI platforms (OpenAI, Ollama, Dify, etc.) and uses Qdrant for vector retrieval to support RAG capabilities. The system supports both local and cloud-based voice processing and implements a multi-device "nearest response" logic to ensure only one device answers when multiple are triggered in the same space.

Who it’s for

  • Users who own ESP32 hardware and need a comprehensive management platform.
  • Developers looking for a stable, scalable backend for AI-powered voice assistants.
  • Organizations needing to support high-concurrency device connections.

Highlights

  • Comprehensive AI Integration: Supports multiple LLM providers and the MCP tool protocol for extended capabilities.
  • Advanced Voice Pipeline: Features real-time streaming interaction, voice cloning, and intelligent interruption handling.
  • Personalized Experience: Includes voiceprint recognition to identify family members and maintain independent memories for each user.
  • Memory Systems: Implements RAG knowledge bases and long-term memory graphs to remember user preferences across sessions.
  • IoT Control: Enables voice-driven smart home control via Function Calling.
  • Enterprise Tooling: Includes full-link monitoring for tokens, latency, and device activity, with one-click Docker deployment.

Related

  • Project
    xinnan-tech/xiaozhi-esp32-server

    A backend server for ESP32-based smart hardware that enables custom voice-controlled AI interactions, device management, and multi-modal processing.

  • Project
    hackers365/xiaozhi-esp32-server-golang

    A high-performance Go-based AI backend for ESP32 devices that enables low-latency, full-streaming voice interactions by integrating ASR, LLM, and TTS engines.

  • Project
    akdeb/ElatoAI

    A framework for deploying real-time voice AI on ESP32 devices, connecting low-power hardware to frontier speech-to-speech models via edge functions.

  • Project
    78/xiaozhi-esp32

    XiaoZhi ESP32 is an open‑source firmware that turns ESP32‑based boards into voice‑interactive AI chatbots. It captures audio, uses ESP‑SR for wake‑word detection, streams to cloud LLMs (Qwen/DeepSeek) via WebSocket or MQTT/UDP, and returns TTS responses. The device also runs the MCP protocol for controlling peripherals and smart‑home actions, supporting many boards, networking options, multilingual UI, speaker ID, and customizable assets.

  • Project
    espressif/esp-skainet

    ESP‑Skainet is Espressif’s on‑device voice‑assistant SDK for ESP32 chips. It bundles a tiny wake‑word engine (WakeNet), an offline command recognizer (MultiNet) supporting up to 200 languages, and a full audio front‑end (AEC, VAD, NS). The stack runs on ESP32‑S3 (or other ESP32 boards) with minimal RAM/flash, enabling privacy‑preserving voice control for smart‑home, wearables, or robotics. The repo includes ready‑to‑flash examples, detailed docs, and support for custom wake words and commands.