huggingface/speech-to-speech

Speech‑to‑Speech (huggingface/speech-to-speech)

What it is – An open‑source, low‑latency pipeline that turns spoken input into spoken output, i.e. a voice‑agent stack. It wires together four stages – Voice‑Activity Detection, Speech‑to‑Text, a Language Model, and Text‑to‑Speech – and exposes the whole thing through the OpenAI Realtime protocol (WebSocket / WebRTC). Every stage can be swapped out for a different model or provider, so you can run everything locally, mix local and hosted components, or point the LLM at any OpenAI‑compatible service (OpenAI, HF Inference Providers, vLLM, llama.cpp, etc.).

Core ideas

Stage	Default backend	What you can replace it with
VAD	Silero VAD v5	Any VAD that can emit turn‑detection events (DeepFilterNet is optional)
STT	Parakeet TDT (local)	Whisper (transformers), Faster‑Whisper, Lightning‑Whisper‑MLX, Paraformer, OpenAI `/v1/audio/transcriptions`, etc.
LLM	OpenAI‑compatible Responses API (default model `gpt‑5.6‑terra`)	Direct `transformers` inference, `mlx‑lm` on Apple Silicon, self‑hosted vLLM/llama.cpp, any OpenAI‑compatible endpoint
TTS	Qwen3‑TTS (GGML on Linux, mlx‑audio on macOS)	Kokoro‑82M, Pocket TTS, ChatTTS, OmniVoice, MMS‑TTS, OpenAI `/v1/audio/speech`, etc.

All components run in separate threads and communicate via queues, which keeps the pipeline responsive and makes it easy to plug in new backends.

Quick start (from the README)

pip install speech-to-speech               # Python 3.10+
export OPENAI_API_KEY=...                  # needed for the default LLM backend
speech-to-speech serve                     # starts a Realtime‑compatible server
# in another terminal
speech-to-speech talk --url ws://127.0.0.1:8765/v1/realtime

serve → runs the VAD‑STT‑LLM‑TTS pipeline and listens on ws://localhost:8765/v1/realtime.
talk → a bundled microphone‑speaker client that talks to that server.
local → runs both server and client in the same process (handy for testing).

You can also point the LLM at a self‑hosted Gemma 4 model with llama.cpp:

llama-server -hf ggml-org/gemma-4-E4B-it-GGUF -np 2 -c 65536 -fa on --swa-full
speech-to-speech serve \
    --model_name "ggml-org/gemma-4-E4B-it-GGUF" \
    --responses_api_base_url "http://127.0.0.1:8080/v1" \
    --responses_api_api_key ""

Main features (as described in the README)

OpenAI Realtime compatibility – Implements the core subset of the Realtime event set, so existing OpenAI Agents SDK code works out‑of‑the‑box.
Fully modular – Each stage is selectable via CLI flags (--stt, --llm_backend, --tts).
Local‑first defaults – Parakeet TDT for STT, Qwen3‑TTS for speech, and an OpenAI‑compatible LLM API, all installable with a single pip install.
Cross‑platform – Works on Linux, macOS, and Windows (via platform‑specific wheels). macOS uses mlx‑audio for TTS; Linux can use GGML or CUDA‑accelerated Qwen3‑TTS.
Optional extras – Extras ([kokoro], [pocket], [omnivoice], etc.) let you pull in alternative TTS or STT models without bloating the base install.
Docker support – docker compose up launches a llama.cpp server with Gemma 4 and the Realtime server together.
LLM proxy – With --enable_llm_proxy the same process also exposes the configured LLM as a plain OpenAI‑compatible /v1/chat/completions or /v1/responses endpoint for side‑tasks.
Tool‑calling support – The packaged client can load a Python module that implements tool calls, enabling richer interactions (e.g., web search via Serper).

Typical use cases

Robotic voice assistants – Used as the conversation backend for thousands of Reachy Mini robots.
Desktop or embedded voice agents – Run entirely locally on a laptop or on‑device hardware (Apple Silicon, CUDA GPUs, or CPU‑only).
Prototyping multimodal agents – Combine the voice pipeline with vision models (mlx‑lm support) or custom tool modules.
Research on low‑latency spoken interaction – The modular design lets you swap in experimental STT/TTS models and measure end‑to‑end latency.

Limitations / gotchas (from the README)

The Realtime implementation covers only the core event set; it is not a full drop‑in replacement for the entire OpenAI Realtime API.
Some optional components conflict (e.g., DeepFilterNet requires numpy<2 while Pocket TTS needs numpy>=2). You must install them in mutually exclusive environments.
CUDA‑accelerated Qwen3‑TTS wheels target CUDA 12.8 and recent glibc; older runtimes need the matching wheel from the provided Hugging Face wheelhouse.
The default LLM (gpt‑5.6‑terra) is accessed via the OpenAI Responses API; you need a valid API key unless you replace it with a self‑hosted model.
Direct audio‑input mode (sending raw VAD audio to an audio‑capable LLM) works only with the chat‑completions backend and requires a model that explicitly supports audio.

Installation snapshot

pip install speech-to-speech                     # core
pip install "speech-to-speech[kokoro]"          # optional Kokoro‑82M TTS
pip install "speech-to-speech[pocket]"          # optional Pocket TTS
# …other extras as needed

For development:

git clone https://github.com/huggingface/speech-to-speech.git
cd speech-to-speech
uv sync                     # creates an editable install

Quick reference table (from the README)

Component	Backends (selected via CLI)	Platform notes
VAD	Silero VAD v5 (built‑in)	works everywhere
STT	Parakeet TDT, Whisper, Faster‑Whisper, Lightning‑Whisper‑MLX, Paraformer, OpenAI `/v1/audio/transcriptions`	CUDA/CPU/Apple‑Silicon variants
LLM	OpenAI‑compatible Responses API, Chat‑Completions API, `transformers`, `mlx‑lm`	can point at remote providers or local vLLM/llama.cpp
TTS	Qwen3‑TTS (default), Kokoro‑82M, Pocket TTS, ChatTTS, OmniVoice, MMS‑TTS, OpenAI `/v1/audio/speech`	GGML/CUDA on Linux, mlx‑audio on macOS

Where to go next?

Read the Realtime Engine README for the exact event matrix and how to integrate with the OpenAI Agents SDK.
Explore the LLM backends section to decide whether you want a hosted API or a fully local model.
Try the Docker compose setup if you want a ready‑made container that runs both a local LLM server and the speech‑to‑speech pipeline.
Check the tool‑calling documentation if you need the voice agent to invoke external services (search, database queries, etc.).

Bottom line – speech-to-speech is a production‑grade, open‑source voice‑agent framework that lets you build end‑to‑end spoken assistants with interchangeable AI components, all behind an OpenAI‑compatible Realtime interface.