Deploying Speech-to-Speech on Hugging Face Inference Endpoints

Hugging Face has introduced a method for deploying its Speech-to-Speech (S2S) pipeline using custom Docker images on Inference Endpoints. This approach allows developers to run a computationally intensive multi-model pipeline with low latency by leveraging scalable GPU infrastructure without managing underlying hardware.

The Speech-to-Speech (S2S) Pipeline

The Speech-to-Speech project implements a cascaded pipeline using models from the Transformers library. The system enables a user to speak and receive a synthesized voice response through four primary components:

  1. Voice Activity Detection (VAD)
  2. Speech to Text (STT)
  3. Language Model (LM)
  4. Text to Speech (TTS)

The pipeline supports English, French, Spanish, Chinese, Japanese, and Korean, and includes an auto flag for automatic language detection.

Deploying via Custom Docker Images

Because S2S requires significant computational resources, Hugging Face recommends using Inference Endpoints (IE) with a custom Docker image rather than pre-built models or custom handlers. This method optimizes performance by encapsulating all dependencies and data within the image.

Building the Custom Image

To create the deployment image, the process begins by cloning the huggingface-inference-toolkit repository to establish a pre-optimized base for inference workloads. The customization process involves:

  • Integrating Code and Data: The S2S codebase and required datasets (such as fast-unidic) are added as git submodules. Including data directly in the container reduces startup time by eliminating the need to download datasets during endpoint instantiation.
  • Streamlining Dependencies: The Dockerfile is modified to remove irrelevant packages and move the installation of requirements.txt from the entry point to the build stage, ensuring dependencies are pre-installed before runtime.

Deployment Configuration

Custom images can be deployed through the Inference Endpoints GUI or API. Key configuration requirements include:

  • Hardware: An AWS GPU L4 instance is recommended (costing approximately $0.80 per hour).
  • Container Settings: The container port must be set to 80 to match the Inference Endpoint expectation, as the default toolkit entrypoint uses port 5000.
  • Security: An HF_TOKEN must be provided as a secret to allow the container to download gated models, such as Meta-Llama-3.1-8B-Instruct.

Technical Architecture for Low Latency

To achieve low latency, the deployment utilizes a custom webserver and a specialized client rather than standard HTTP requests.

WebSocket Webserver

The webserver, implemented using Starlette, supports WebSocket connections to facilitate streaming audio. The server runs a prepare_handler on startup to initialize and warm up models. During operation, the inference_handler.process_streaming_data method receives audio, chunks it for the VAD, and manages a queue for processing and returning the synthesized voice response.

Audio Streaming Client

A dedicated client handles the interface between the user's hardware and the webserver. Its responsibilities are divided into four primary tasks:

  1. Recording: Capturing audio via an audio_input_callback and submitting chunks to a queue.
  2. Submission: Sending audio to the server via the send_audio method.
  3. Reception: Receiving server responses through an on_message method, which determines if the server has finished responding or if more data should be added to the playback queue.
  4. Playback: Playing back the audio responses via an audio_output_callback, ensuring the audio is within safe ranges to prevent hardware damage.

Sources