SpeechT5: Unified-Modal Encoder-Decoder for Speech Processing

Overview

SpeechT5 is a unified-modal encoder-decoder model designed to handle multiple spoken language processing tasks within a single architecture. By pre-training on a mixture of text-to-speech, speech-to-text, text-to-text, and speech-to-speech data, the model develops a unified space of hidden representations shared by both text and speech.

Core Architecture

SpeechT5 utilizes a standard Transformer encoder-decoder backbone for all tasks. To enable the processing of diverse data types, the model employs modular "pre-nets" and "post-nets":

  • Pre-nets: Convert input text or speech into the hidden representations required by the Transformer encoder.
  • Post-nets: Convert the Transformer's output hidden representations back into text or speech.

While the model is pre-trained using all pre-nets and post-nets simultaneously, it is fine-tuned for specific tasks. Once fine-tuned, a model uses only the pre-nets and post-nets relevant to that task. Because the fine-tuning process alters the weights, a model fine-tuned for one task (e.g., ASR) cannot be converted to another (e.g., TTS) simply by swapping the pre- and post-nets.

Text-to-Speech (TTS) Capabilities

SpeechT5 serves as a versatile text-to-speech system that can synthesize audio for multiple speakers using speaker embeddings.

Technical Implementation

For the TTS task, SpeechT5 uses a specific set of modules:

  • Text encoder pre-net: A text embedding layer mapping tokens to hidden representations.
  • Speech decoder pre-net: A sequence of linear layers that compress a log mel spectrogram into hidden representations (based on Tacotron 2).
  • Speech decoder post-net: A module that predicts a residual to refine the output spectrogram (based on Tacotron 2).

Audio Generation Workflow

To produce a final waveform, the model generates a log mel spectrogram, which must then be processed by a vocoder. Hugging Face provides a HiFi-GAN based vocoder (SpeechT5HifiGan) for this purpose. The system operates at a constant sample rate of 16 kHz.

Speech-to-Speech and Voice Conversion

SpeechT5 can perform speech-to-speech tasks, such as voice conversion, by replacing the text encoder pre-net with a speech encoder pre-net.

Technical Implementation

  • Speech encoder pre-net: This module uses convolution layers to downsample input waveforms into audio frame representations, utilizing the same feature encoding module found in wav2vec 2.0.

Voice conversion is achieved by providing the model with an input speech waveform and a target speaker embedding, allowing the model to transform the original voice into the characteristics of the target speaker.

Automatic Speech Recognition (ASR)

SpeechT5 provides speech-to-text capabilities for transcription and speaker identification.

Technical Implementation

For ASR, the model employs:

  • Speech encoder pre-net: The same CNN feature encoder used in the speech-to-speech model (wav2vec 2.0).
  • Text decoder pre-net: An embedding layer mapping text tokens to hidden representations.
  • Text decoder post-net: A single linear layer that projects hidden representations into probabilities over the vocabulary.

Because the tokenizer operates at the character level, the resulting ASR transcriptions do not include punctuation or capitalization.

Summary of Versatility

Beyond the integrated ASR, TTS, and voice conversion checkpoints, the original research indicates that the SpeechT5 architecture is capable of speech translation, speech enhancement, and speaker identification.

Sources