Qwen2.5-Omni Release: End-to-End Multimodal Model for Real-Time Interaction

Qwen has announced the release of Qwen2.5-Omni, a flagship end-to-end multimodal model designed for comprehensive perception across text, images, audio, and video. The model enables real-time streaming responses through both text generation and natural speech synthesis, providing a cohesive interface for multimodal interaction.

Thinker-Talker Architecture

Qwen2.5-Omni utilizes a novel "Thinker-Talker" architecture to achieve seamless multimodal processing and generation. This architecture splits the functional roles of the model into two primary components:

  • Thinker: Acting as the central processing unit, the Thinker is a Transformer decoder supported by audio and image encoders. It processes inputs from text, audio, and video modalities to generate high-level representations and text.
  • Talker: Serving as the output mechanism, the Talker is a dual-track autoregressive Transformer Decoder. It receives high-dimensional representations and text from the Thinker in a streaming manner and outputs discrete speech tokens fluidly.

Because the Talker shares all historical context information from the Thinker, the system operates as a single cohesive model, allowing for end-to-end training and inference.

Time-aligned Multimodal RoPE (TMRoPE)

To synchronize the timestamps of video inputs with audio, Qwen2.5-Omni introduces TMRoPE (Time-aligned Multimodal RoPE), a novel position embedding designed specifically for multimodal synchronization.

Capabilities and Performance

Qwen2.5-Omni is designed for real-time voice and video chat, supporting chunked input and immediate output. It demonstrates strong performance across multiple modalities, often rivaling similarly sized single-modality models.

Multimodal Integration

In tasks requiring the integration of multiple modalities, Qwen2.5-Omni achieves state-of-the-art performance on the OmniBench benchmark.

Single-Modality Performance

The model exhibits high proficiency in several specialized areas:

  • Audio: It outperforms the similarly sized Qwen2-Audio model in audio capabilities and shows strong results in speech recognition (Common Voice), translation (CoVoST2), and audio understanding (MMAU).
  • Visual: It achieves performance comparable to Qwen2.5-VL-7B in image reasoning (MMMU, MMStar) and video understanding (MVBench).
  • Speech Generation: The model demonstrates superior robustness and naturalness in speech generation, as evaluated by Seed-tts-eval and subjective naturalness tests.
  • Instruction Following: End-to-end speech instruction following performance rivals the model's effectiveness when using text inputs, as evidenced by benchmarks such as MMLU and GSM8K.

Future Development

Qwen intends to further enhance Qwen2.5-Omni's ability to follow voice commands and improve collaborative audio-visual understanding. The lab also aims to integrate additional modalities to move closer to a fully omni-model.

Sources