Automatic Speech Recognition for Large Files with Wav2Vec2 in Transformers

Hugging Face has introduced a method to perform high-quality automatic speech recognition (ASR) on arbitrarily long audio files and during live inference by leveraging the specificities of the Connectionist Temporal Classification (CTC) architecture in Wav2Vec2.

The Challenge of Sequence Length in Transformers

Wav2Vec2, a popular pre-trained model for speech recognition released by Meta AI Research, is based on the Transformer architecture. A fundamental limitation of Transformers is their finite sequence length capacity, primarily due to the $O(n^2)$ complexity of the attention mechanism relative to the sequence length.

Attempting to run Wav2Vec2 on very long audio files (e.g., an hour-long recording) without chunking will result in memory exhaustion and program crashes, even on high-end hardware like the NVIDIA A100 GPU.

Simple Chunking and Its Limitations

Simple chunking involves dividing an audio file into shorter, fixed-length samples (e.g., 10 seconds each), performing inference on each, and reconstructing the final text. While computationally efficient, this approach often produces subpar results because the model lacks necessary context at the chunk boundaries, leading to poor inference quality where the audio is split.

Common attempts to mitigate this—such as chunking only during silence or using a separate voice activity detection model—are not entirely robust, as audio may contain continuous speech or long periods of noise.

Chunking with Stride

Wav2Vec2 utilizes the CTC algorithm, where every frame of audio is mapped to a single letter prediction (logit). Hugging Face leverages this property to implement "chunking with stride," which allows for robust ASR on long files:

  1. Overlapping Chunks: Inference is performed on chunks that overlap. This ensures the model has sufficient context in the center of each chunk.
  2. Logit Dropping: The logits at the edges of the chunks—where inference quality is typically lowest—are dropped.
  3. Logit Chaining: The remaining central logits are chained together to reconstruct a transcription that closely approximates the result of running the model on the full-length audio.

In the transformers pipeline, this is enabled by adding the chunk_length_s argument. Users can further customize the overlap using stride_length_s, which accepts a tuple for left and right stride lengths (by default, the stride is 1/6th of the chunk length on each side).

Compatibility with Language Model (LM) Augmentation

Support for adding a Language Model (LM) to Wav2Vec2 to improve Word Error Rate (WER) without additional fine-tuning is also compatible with this striding technique. Because the LM operates directly on the logits, the chunking and striding process can be applied to LM-boosted models without modification.

Live Inference Capabilities

Because Wav2Vec2 is a single-pass CTC model, it is highly efficient, particularly on GPUs. This efficiency enables live inference by feeding data into the pipeline as it arrives. By applying striding to continuous chunks (e.g., 10-second chunks with 1-second striding), the model can provide real-time transcriptions as a user speaks, eliminating the need to wait for a full chunk to be processed before displaying text.

Sources