akdeb/ElatoAI

Realtime Voice AI with 100+ Models on Arduino ESP32 with Secure Websockets and Edge Functions for AI Companions, and Devices

What it solves

ElatoAI enables the creation of real-time voice AI devices using affordable ESP32 hardware. It bridges the gap between high-level AI speech models (like OpenAI's Realtime API or Gemini Live) and low-power IoT devices, allowing users to build physical AI toys or assistants that can engage in long, uninterrupted voice conversations without needing expensive onboard processing.

How it works

The system uses a three-tier architecture:

  1. ESP32 IoT Client: A hardware device that captures audio, compresses it using the Opus codec, and sends it via Secure WebSockets to an edge server.
  2. Edge Server Functions: Hosted on Deno Edge or Cloudflare Workers, these servers act as a proxy that manages WebSocket connections and communicates with various AI providers (OpenAI, Gemini, xAI, Eleven Labs, etc.).
  3. Frontend Client: A Next.js web application used to configure AI agents, manage device authentication, and control hardware settings like volume and voice pitch.

Who it’s for

  • IoT developers and hobbyists wanting to integrate frontier voice AI into physical hardware.
  • Makers building AI-powered toys or interactive gadgets.
  • Developers experimenting with low-latency speech-to-speech pipelines on edge infrastructure.

Highlights

  • Broad Model Support: Integrates with OpenAI Realtime, Gemini Live, xAI Grok, Eleven Labs, Hume AI, and Boson Higgs.
  • Optimized Audio: Uses Opus compression for high-quality audio streaming at low bandwidth (12kbps).
  • Edge Performance: Leverages Deno and Cloudflare Workers to maintain low round-trip latency globally.
  • Hardware Friendly: Designed for ESP32-S3 and does not require PSRAM to function.
  • Advanced Device Features: Supports OTA (Over-the-Air) updates, captive portal for WiFi management, and tool calling for agentic capabilities.

Related

  • Project
    hackers365/xiaozhi-esp32-server-golang

    A high-performance Go-based AI backend for ESP32 devices that enables low-latency, full-streaming voice interactions by integrating ASR, LLM, and TTS engines.

  • Project
    xinnan-tech/xiaozhi-esp32-server

    A backend server for the xiaozhi-esp32 project that enables ESP32 devices to become AI assistants with voice, vision, and smart home control capabilities.

  • Project
    espressif/esp-claw

    An AI agent framework for ESP32-series IoT devices that enables users to define device behavior through chat and execute decision-making locally on the edge.

  • Project
    espressif/esp-sr

    A speech recognition framework for Espressif chips that enables offline wake word detection, voice command recognition, and audio preprocessing.

  • Project
    78/xiaozhi-esp32

    An open-source AI chatbot framework for ESP32 microcontrollers that enables voice interaction with LLMs and hardware control via the MCP protocol.