hackers365/xiaozhi-esp32-server-golang

golang版本的小智后端服务，支持websocket和mqtt+udp协议，支持声纹识别/声音克隆/知识库/mcp远程调用/主动音频下发/openclaw等功能

What it solves

It provides a high-performance, full-streaming AI backend specifically designed for IoT and smart voice scenarios, enabling ESP32 devices and other edge terminals to perform real-time AI voice interactions with low latency.

How it works

The server is built with Go and implements an end-to-end streaming pipeline: Voice Activity Detection (VAD) $\rightarrow$ Automatic Speech Recognition (ASR) $\rightarrow$ Large Language Model (LLM) $\rightarrow$ Text-to-Speech (TTS). It uses a transport abstraction layer to support multiple protocols (WebSocket, MQTT, UDP) and employs message queues for asynchronous processing of LLM and TTS tasks. It integrates various AI engines via the Eino framework and includes a web-based management console for configuration and monitoring.

Who it’s for

Developers building AI-powered voice assistants or smart home devices using ESP32 hardware who need a scalable, modular backend to handle voice processing and agent logic.

Highlights

  • Full-Streaming Pipeline: Low-latency real-time interaction from speech input to audio output.
  • Speaker Identification: Automatically switches TTS voices based on the identified speaker's identity.
  • Modular Architecture: Plug-and-play modules for VAD, ASR, LLM, TTS, and vision capabilities.
  • Broad Integration: Supports multiple engines including FunASR, OpenAI, Ollama, Doubao, EdgeTTS, and CosyVoice.
  • Advanced AI Features: Includes RAG support via Dify/RAGFlow, voice cloning, and MCP (Model Context Protocol) for tool calling and resource management.
  • Comprehensive Management: A web console for visual configuration, latency monitoring, and OTA verification.

Related

  • Project
    xinnan-tech/xiaozhi-esp32-server

    A backend server for the xiaozhi-esp32 project that enables ESP32 devices to become AI assistants with voice, vision, and smart home control capabilities.

  • Project
    akdeb/ElatoAI

    A framework for deploying real-time voice AI models on ESP32 devices, enabling low-latency speech-to-speech interactions via edge functions.

  • Project
    huggingface/speech-to-speech

    Speech‑to‑Speech is a Hugging Face open‑source pipeline that turns spoken input into spoken output. It chains VAD → STT → LLM → TTS, each component being swappable (Silero VAD, Parakeet/Faster‑Whisper/Whisper STT, any OpenAI‑compatible LLM or local Transformers/MLX model, Qwen3‑TTS or other TTS back‑ends). The whole system speaks the OpenAI Realtime protocol over WebSocket/WebRTC, so existing OpenAI Agents SDK code works. Install with a single `pip install speech-to-speech`, then run `speech-to-speech serve` (server) and `speech-to-speech talk` (client) or `speech-to-speech local` (both together). It supports local‑only operation, mixed local/hosted LLMs, Docker deployment, and optional extras for alternative TTS/STT models. Designed for voice‑assistant robots (e.g., Reachy Mini) and low‑latency spoken AI applications.

  • Dispatch
    Reachy Mini Local Speech Backend Integration

    Hugging Face has released a local speech-to-speech pipeline for Reachy Mini, allowing the robot to handle conversations fully locally using a cascaded VAD, STT, LLM, and TTS stack.

  • Project
    espressif/esp-sr

    ESP‑SR is Espressif’s on‑device speech‑recognition framework for ESP32 MCUs. It bundles a tiny audio front‑end, wake‑word detection (WakeNet), voice‑activity detection (VADNet), offline command spotting (MultiNet), and speech synthesis, all optimized for low‑memory, low‑power chips. The library is modular, supports multiple languages, and lets developers add voice control to IoT devices without needing cloud services.