Qwen3.5-LiveTranslate-Flash Release Notes

Qwen has announced Qwen3.5-LiveTranslate-Flash, a simultaneous interpretation model built on the Qwen3.5-Omni architecture. The model enables real-time, multimodal translation by integrating audio and visual context to improve translation accuracy and identity consistency through cross-lingual voice cloning.

Key Technical Upgrades

Qwen3.5-LiveTranslate-Flash introduces significant improvements in language coverage, latency, and terminology handling compared to its predecessor, Qwen3-LiveTranslate.

Expanded Language Support

The model has massively expanded its linguistic capabilities:

  • Input audio and output text: Increased from 18 to 60 languages.
  • Output audio: Increased from 10 to 29 languages.

Ultra-Low Latency via Readable Unit Technology

To minimize delays in simultaneous interpretation, the model utilizes a novel "Readable Unit" real-time translation technique. This strategy allows for more aggressive streaming output while maintaining semantic consistency and readability.

Performance benchmarks show that Qwen3.5-LiveTranslate-Flash reduces first-token latency by 3.45 seconds and per-token latency by 1.88 seconds compared to the previous version, achieving an average speech-to-speech per-token latency of 2.8 seconds.

Real-Time Voice Cloning

The system can replicate a speaker's vocal characteristics from a single sentence, allowing the translated speech to sound like the original speaker across different languages. This feature supports three modes: pre-registered, clone-once, and real-time.

Dynamic Hotword Enhancement

To reduce mistranslation of proper nouns and industry-specific terms, the model includes a built-in Hotword capability. Users can dynamically configure and update names, places, brand names, and product models in real time per scenario.

Model Architecture

Qwen3.5-LiveTranslate is based on the Qwen3.5-Omni Thinker-Talker architecture:

  • The Thinker: Processes interleaved visual and audio inputs to generate text translations.
  • The Talker: Uses the translated text and source audio to produce speech with cross-lingual voice cloning.

For real-time interpretation, the model employs a chunk-wise streaming input mechanism and Readable Unit tags to control the granularity of speech synthesis.

Multimodal Capabilities and Use Cases

Qwen3.5-LiveTranslate-Flash leverages visual context to resolve translation ambiguities, using on-screen text or objects in the scene to select the correct meaning of a word or phrase.

Key application scenarios include:

  • International Meetings: Handling code-switching, diverse accents, and domain-specific terminology in real time.
  • Travel: Integrating with AI Glasses to combine visual context (e.g., menus) with spoken dialogue for on-device translation.
  • Livestreaming: Precise translation of product specifications and numerical data for e-commerce localization.
  • Literary Translation: Translating archaic prose, such as classical Chinese (文言文) from the Romance of the Three Kingdoms, into modern English.

Implementation via DashScope API

The model is accessible via the DashScope API using a WebSocket connection (wss://dashscope.aliyuncs.com/api-ws/v1/realtime). The API supports:

  • Modalities: Configurable output for both text and audio or text-only.
  • Input: PCM audio chunks and image frames for visual context.
  • Customization: A corpus field for registering hotwords to boost accuracy on specific phrases.

Future Development Directions

Qwen intends to further evolve the model by focusing on:

  • Further reducing end-to-end latency toward real-time limits.
  • Expanding coverage for low-resource languages and regional dialects.
  • Improving consistency for terminology and names in long-form dialogues.
  • Enhancing voice cloning fidelity to include ambient sounds and on-site atmosphere.
  • Integrating joint multimodal modeling for gestures, lip movements, and expressions.

Sources