Hugging Face Inference Endpoints: Fast Whisper Transcriptions

Hugging Face has launched a new deployment option for OpenAI Whisper on Inference Endpoints, providing up to 8x performance improvements over the previous version. This update enables users to deploy dedicated, cost-effective transcription models using a community-centric approach that leverages open-source optimizations.

Optimized Inference Stack

The new Whisper endpoint is powered by the vLLM project, allowing for efficient model execution across various hardware families, specifically targeting NVIDIA GPUs with compute capabilities 8.9 or better (Ada Lovelace), such as the L4 and L40s.

To achieve these performance gains, the stack employs three primary software optimizations:

  • PyTorch Compilation (torch.compile): Generates optimized kernels in a Just-In-Time (JIT) fashion, allowing the system to modify the computational graph and reorder operations.
  • CUDA Graphs: Reduces GPU scheduling overhead, data movements, and synchronizations by recording sequential operations and grouping them into larger work units.
  • Float8 KV Cache: Dynamically quantizes activations to reduce memory requirements. While computations are performed in bfloat16 (half precision), outputs are stored in float8 (1 byte vs 2 bytes for bfloat16), increasing the cache hit rate by allowing more elements to be stored in the KV cache.

Performance Benchmarks

Evaluations were conducted on a single L4 GPU using bfloat16 with consistent decoding settings for language, beam size, and batch size. The benchmarks focused on three models: Whisper Large V3, Whisper Large V3-Turbo, and Distil-Whisper Large V3.5.

Transcription Quality

Transcription accuracy was measured using Word Error Rate (WER) across eight standard datasets from the Open ASR Leaderboard, including AMI, GigaSpeech, LibriSpeech (Clean and Other), SPGISpeech, Tedlium, VoxPopuli, and Earnings22. The results indicate that all three Whisper variants maintain WER performance comparable to their Transformers library baselines, meaning there is no loss in transcription quality despite the speed increases.

Runtime Efficiency

Using the rev16 long-form dataset (audio segments over 45 minutes), Hugging Face measured the Real-Time Factor (RTFx)—the ratio of audio duration to transcription time. Whisper Large V3 showed a nearly 8x improvement in RTFx compared to previous implementations.

Deployment and Implementation

Users can deploy ASR (Automatic Speech Recognition) inference pipelines via Hugging Face Endpoints by selecting a model and configuring parameters.

Python Implementation Example

Inference can be performed using a simple HTTP request to the deployed endpoint. Below is the implementation for testing a deployed checkpoint:

import requests

ENDPOINT_URL = "https://<your‑hf‑endpoint>.cloud/api/v1/audio/transcriptions"
HF_TOKEN     = "hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
AUDIO_FILE   = "sample.wav"

headers = {"Authorization": f"Bearer {HF_TOKEN}"}

with open(AUDIO_FILE, "rb") as f:
    files = {"file": f.read()}

response = requests.post(ENDPOINT_URL, headers=headers, files=files)
response.raise_for_status()

print("Transcript:", response.json()["text"])

Real-Time Applications

The increased speed of these endpoints enables the development of real-time transcription applications. Hugging Face highlighted this capability through a demo built with FastRTC, allowing users to transcribe speech from a microphone in real time.

Sources