pollen-robotics/reachy_mini_conversation_app

Reachy Mini Conversation App – What It Is

Purpose – A Python application that lets you talk to the Reachy Mini robot. It streams your voice to a Hugging Face realtime speech model, sends the transcribed text to a large‑language‑model (LLM) assistant, and turns the assistant’s replies into robot actions (dances, head movements, emotions, etc.).

Key capabilities

Low‑latency voice – uses Hugging Face’s realtime backend (or a locally‑hosted endpoint) for speech‑to‑text and text‑to‑speech.
Vision integration – the camera tool can capture a frame and feed it to the same realtime backend for image analysis.
Motion blending – a queue system handles high‑level moves (dances, poses, breathing) while a “wobble” layer reacts to speech rhythm.
Tool ecosystem – the LLM can call a set of tools (e.g., dance, move_head, remember, web‑search, weather) that are either local Python modules or remote Hugging Face Spaces.
Personality profiles – each profile defines a prompt, voice, greeting, and default tools; you can switch personalities on the fly via the optional web UI.
Long‑term memory – optional persistent facts about the user that the assistant can recall across sessions.
Local VAD & barge‑in – optional Silero voice‑activity‑detection runs on‑device for faster interruption handling and smoother robot listening.

How It Works (Architecture at a Glance)

User speaks → microphone audio (16 kHz) is sent to the Hugging Face realtime service.
The service returns a transcription and, optionally, a synthesized reply.
The transcribed turn is fed to the LLM assistant, which may request a tool.
The app executes the tool (e.g., move the robot’s head, play an emotion clip, query the web) and returns the result to the LLM.
The LLM’s final answer is spoken back to the user and may trigger motion blending.

A diagram (conversation_app_arch.svg) in the repo visualises these components.

Getting Started

Prerequisites

Reachy Mini SDK must be installed first (see the linked repo).
Python 3.12 and either uv (recommended) or pip for dependency management.
Optional: a local Hugging Face realtime endpoint if you don’t want to use the hosted service.

Installation (uv example)

uv venv --python /opt/homebrew/bin/python3.12 .venv
source .venv/bin/activate
uv sync               # install runtime deps
uv sync --group dev  # install dev tools (optional)

(Or use the classic python -m venv + pip install -e . workflow.)

Configuration

Copy .env.example → .env and adjust variables such as:

HF_REALTIME_CONNECTION_MODE – deployed (default) or local.
HF_REALTIME_WS_URL – websocket URL for a self‑hosted backend.
LOCAL_VAD_ENABLED / LOCAL_BARGE_IN_ENABLED – toggle on‑device voice detection and interruption.
REACHY_MINI_MEMORY_ENABLED – turn persistent memory on/off.

Settings edited in the web UI are persisted to settings.json and override the .env defaults.

Running the App

reachy-mini-conversation-app          # console‑only mode
reachy-mini-conversation-app --ui    # also start the web UI (http://127.0.0.1:7860/)

Make sure the Reachy Mini daemon is running first; otherwise you’ll see a TimeoutError.

CLI flags let you disable the camera (--no-camera), pick a specific robot on a network (--robot-name), or enable verbose debugging (--debug).

Tools – What the Assistant Can Do

Tool	What it does	Where it lives
`dance` / `stop_dance`	Queue or clear dance sequences from the public dances library dataset.	Core install
`play_emotion` / `stop_emotion`	Play a short emotion audio clip from the emotions library dataset.	Core install
`camera`	Capture a frame and run it through the realtime backend (requires a camera).	Core install
`move_head` / `head_tracking` / `sweep_look`	Control head pose or follow a face.	Core install
`remember` / `forget`	Store or delete a short fact about the user (persisted in `memory.v1.json`).	Core install
`volume_control`	Get/set speaker or mic volume via the robot daemon.	Core install
`robot_status`	Query robot metadata (software version, Wi‑Fi IP, IMU, installed apps).	Core install
`pollen_robotics_reachy_mini_search_tool__search_web`	Web search via a pre‑published Hugging Face Space.	Remote Space
`pollen_robotics_reachy_mini_weather_tool__get_weather`	Current weather lookup.	Remote Space
`pollen_robotics_reachy_mini_time_tool__get_time`	Time‑zone aware clock info.	Remote Space
`pollen_robotics_reachy_mini_knowledge_tool__*`	Company/product FAQ and documentation search.	Remote Space

You can add local tools by creating a Python module that subclasses Tool and placing it in src/reachy_mini_conversation_app/tools/. Remote tools are added by installing a Hugging Face Space that implements the MCP API.

Custom Profiles & Personalities

Profiles live under profiles/ (or a custom directory via REACHY_MINI_CUSTOM_PROFILE). Each profile contains a profile.md file with:

Prompt (Markdown body) that shapes the assistant’s persona.
Optional metadata (voice, greeting, default_tools).
hidden flag to hide it from the UI.

The UI lets you create, select, and edit profiles, and to enable/disable individual tools per personality.

Advanced Topics

Local VAD: Silero VAD model v6.2.1 is downloaded on first run, cached, and run with ONNX Runtime for sub‑second detection.
Barge‑in: When enabled, the robot lowers its playback volume if it detects the user speaking over its response, allowing smooth interruption.
Debug diagnostics: --debug exposes /static/vad-monitor.html and a /rpc WebSocket that streams VAD probabilities and state changes (useful for tuning latency).
Multiple robots: Use --robot-name to address a specific daemon on the same subnet.

Who Might Use This?

Researchers building embodied conversational agents.
Educators demonstrating multimodal AI (speech, vision, motion) on a physical robot.
Hobbyists who want a plug‑and‑play voice‑controlled robot with extensible tool plugins.

License

The repository is released under the MIT License (see LICENSE).