Google DeepMind SL2T: Sign-Language-to-Text Translation for Pixel 11

Google DeepMind has introduced sign-language-to-text (SL2T), a translation model that enables Deaf and hard of hearing users to use sign language as a primary input method on mobile devices. SL2T powers new sign-to-text dictation features in Gboard and Live Transcribe on the Pixel 11, allowing users to sign to their phones to search the web, draft messages, or interact with Gemini, starting with American Sign Language (ASL) to English.

Technical Architecture and Privacy

SL2T utilizes a coordinate-based translation system to ensure user privacy and processing efficiency. Instead of processing raw camera feeds, the system employs an on-device model called MediaPipe Holistic to track pose landmark locations on the signer's body. Only these geometric coordinates are sent to the server for translation, and the original video is discarded immediately.

To improve translation quality, SL2T bypasses "glosses"—the intermediate annotations often used in prior sign language AI. By translating coordinate sequences directly into text, the model better captures non-linear aspects of sign languages, such as spatial constructions and non-manual markers, removing artificial vocabulary limits.

Training and Performance

SL2T is a massively multilingual model trained on over 100,000 hours of data across more than 50 sign languages, with approximately 25% of that data being ASL. The team found that joint training across diverse languages, dialects, and proficiency levels allowed the model to learn shared underlying structures, resulting in better performance than single-language models.

In terms of benchmarks, SL2T achieved a zero-shot score of 70 BLEURT on the FLEURS-ASL (sd-test) benchmark for ASL to English translation, which Google DeepMind reports as significantly higher than any previously reported score.

Real-World Application and Usability

Beyond academic benchmarks, the development of SL2T focused on several practical usability challenges:

• Streaming Latency: Minimizing the delay between signing and text output.
• Input Filtering: Preventing hallucinations when the user is not signing.
• Inclusivity: Ensuring fairness for left-handed signers (approximately 10% of the population) and optimizing performance for one-handed signing, which is common when holding a smartphone.

Community Collaboration and Governance

Google DeepMind developed SL2T through a participatory governance model involving the Deaf community. This included conceptualization by Sam Sepah, a Deaf Googler, and the establishment of the AI Sign Language Advisory Committee (AISLAC), which consists of global Deaf organizations and subject-matter experts.

This collaboration resulted in a joint impact report for the release of SL2T 1.0, which transparently outlines the technology's current capabilities and limitations.

Current Limitations

Based on FLEURS-ASL benchmark examples, the model still exhibits occasional errors in specific areas:

• Rapid fingerspelling: For example, translating "prey" as "grey".
• Classifier depictions: Dropping specific details, such as "claws".
• Tense and Passive Constructions: Occasional errors in tense without sufficient context or in passive sentence structures.