huggingface/speech-to-speech

What it is

speech‑to‑speech is an open‑source Python package that lets you run a complete voice‑assistant locally, without any cloud API keys. It wires together four stages – voice activity detection, speech‑to‑text, a language model, and text‑to‑speech – each of which can be swapped for alternative models. The default pipeline uses Silero VAD, Parakeet TDT for transcription, the Qwen‑3.5‑4B LLM, and Qwen‑3 TTS for audio output.

How it works

1. VAD – Detects when you start and stop speaking so the system knows when to listen.
2. STT – Converts the spoken turn into text (supports many back‑ends such as Parakeet, Whisper, Faster‑Whisper, etc.).
3. LLM – Generates a textual response, streaming tokens and optional tool calls.
4. TTS – Turns the LLM’s reply into spoken audio (default Qwen‑3‑TTS, with many optional TTS models).

Each component runs in its own thread and communicates via queues, making it easy to replace any part with a different model or a remote service.

Main features

• Fully local operation after the first run (no internet required).
• OpenAI Realtime protocol compatibility – the server speaks the same WebSocket/WebRTC API that OpenAI’s Realtime service uses, so existing clients can point at your machine.
• Cross‑platform support: Apple Silicon (MLX‑based models), NVIDIA GPUs (CUDA), or CPU‑only fallback.
• Extensible – dozens of optional back‑ends for VAD, STT, LLM, and TTS can be installed via pip extras.
• CLI commands:
  • speech-to-speech local – runs server + microphone client together.
  • speech-to-speech serve – starts only the Realtime server.
  • speech-to-speech talk – runs a packaged microphone/speaker client against any server.
• Docker support for quick containerised deployment.
• Optional viseme output for avatar mouth animation.

Getting started (quick summary)

python3 -m venv .venv && source .venv/bin/activate
pip install speech-to-speech
speech-to-speech local   # talks to you via your speakers & mic

The first run downloads the required models; subsequent runs are offline.

Who might use it

• Robotics developers building voice‑enabled robots (e.g., the Reachy Mini robot already uses this pipeline).
• App developers who want a drop‑in voice assistant without paying for cloud services.
• Researchers experimenting with different STT/LLM/TTS combos in a modular framework.
• Privacy‑conscious users who need a voice assistant that never leaves the device.

Limitations to be aware of

• Real‑time performance depends on hardware; on CPU‑only machines the LLM will be slow.
• Some optional components (e.g., DeepFilterNet) have version conflicts and must be installed manually.
• The server does not enforce authentication; if exposed beyond a trusted network you need an external gateway.

---

In short: speech‑to‑speech provides a ready‑to‑run, locally hosted voice‑assistant stack that mimics OpenAI’s Realtime API while letting you swap in any open‑source VAD, transcription, language‑model, or TTS component you prefer.