Embedding Gemma Transcribe: High-Speed On-Device Speech-to-Text

Core Innovation: Repurposing Multimodal Embeddings for Transcription

Embedding Gemma Transcribe achieves high-speed, on-device speech-to-text by utilizing the shared latent space of Google's EmbeddingGemma 2 model. Because the model is designed to place audio and text in the same vector space, the system maps short slices of audio directly to the closest match among approximately 229,000 English words. A secondary, lightweight model then applies punctuation and capitalization to the resulting output. This approach allows for transcription speeds of one hour of audio in approximately 13 seconds on a base M3 MacBook Air.

Technical Implementation

The system operates by keeping the original audio encoder of EmbeddingGemma 2 intact while training a small reader component to bridge the gap between audio slices and text tokens. The training process for this reader is minimal, requiring under two hours on 8xH100 GPUs using openly licensed English speech data. The base model itself was not explicitly trained for speech-to-text tasks, demonstrating the efficacy of leveraging existing multimodal embedding spaces for downstream applications.

Performance and Benchmarks

Running on the Apple Neural Engine, the model demonstrates significant throughput advantages over traditional speech-to-text architectures. On a base M3 MacBook Air with 8GB of RAM, the model processes audio at a rate of 288x real-time. Current performance metrics show a Word Error Rate (WER) of approximately 5%, which the developers aim to improve through further training of the reader component.

Model	Hardware	Speed (Real-time Factor)
Embedding Gemma Transcribe	M3 MacBook Air (Neural Engine)	288x
IBM Granite Speech 5.0 TurboCTC	M3 MacBook Air (GPU/MLX)	150x

Community Insights and Observations

Discussion within the developer community highlights both the technical potential and the current limitations of this approach:

• Language Support: The current implementation is optimized for English. Users have noted that while it can attempt to transcribe other languages (such as Portuguese), the quality is significantly lower, and there is high interest in expanding support to multilingual datasets.
• Generalization Concerns: Some observers have questioned the model's ability to handle out-of-vocabulary words, given that the architecture relies on matching audio slices to a fixed set of 229,000 words.
• Hardware Optimization: Experienced users noted that converting models to CoreML for the Apple Neural Engine is a highly effective strategy for increasing inference speed on macOS, often yielding order-of-magnitude improvements compared to CPU-based execution.
• Development Context: The project creator, Shahir, emphasized that the tool was an experimental proof-of-concept for exploring shared vector spaces. Future iterations are expected to focus on improving accuracy and potentially adding Text-to-Speech (TTS) capabilities.

"An order of one or two magnitude of an already existing thing stops becoming a quantity upgrade but rather becomes a quality upgrade in it of itself as it provides access to workflows which previously wouldn't have been feasible."

This sentiment reflects the broader community reaction to the rapid evolution of local, high-speed transcription tools, which have transitioned from unusable to highly efficient in a relatively short period.