Qwen3.5-Omni release notes

TL;DR

Qwen released Qwen3.5‑Omni, a new generation of fully omnimodal large language models that natively handle text, images, audio, and video, support 256k token context, and add multilingual speech recognition (113 languages) and synthesis (36 languages) along with real‑time interaction features such as semantic turn‑taking, web‑search, function calling, voice control, and voice cloning.

Overview

Qwen3.5‑Omni is the latest iteration of Qwen’s omnimodal family. It expands the modality scope of the previous Qwen3‑Omni by pre‑training on massive text, visual, and more than 100 million hours of audio‑visual data. The model family comprises three instruction‑tuned sizes—Plus, Flash, and Light—all of which accept up to 256 k tokens of context. Qwen3.5‑Omni can ingest over 10 hours of raw audio or 400 seconds of 720p video at 1 fps in a single request.

Architecture

The Thinker‑Talker design from Qwen3‑Omni is retained and refined:

  • Thinker receives visual frames via a Vision Encoder and audio via the AuT module. Audio‑visual streams are interleaved with positional encoding using TMRoPE. The Thinker processes the fused omnimodal representation and generates textual output.
  • Talker consumes the Thinker’s text and multimodal cues to produce speech. Speech tokens are encoded with RVQ (Residual Vector Quantization) instead of the heavier DiT blocks, enabling efficient streaming.
  • Both Thinker and Talker employ a Hybrid‑Attention Mixture‑of‑Experts (MoE) layer, which scales capacity while keeping inference cost manageable.
  • Real‑time interaction is achieved through chunk‑wise streaming input for the Thinker and a streaming Talker that interleaves text and speech tokens using ARIA (Adaptive Rate Interleave Alignment). ARIA dynamically aligns token rates to eliminate omissions, misreadings, and unstable pronunciation of numbers.

Multimodal Capabilities

Capability Details
Modalities Text, images, raw audio, and audio‑visual streams (video + audio).
Context Length Up to 256 k tokens (≈ 400 k characters).
Audio Processing Handles > 10 h of continuous audio; supports speech‑to‑text in 113 languages/dialects.
Audio‑Visual Processing Accepts 720p video at 1 fps for up to 400 seconds; can generate structured, timestamped captions and screenplay‑level descriptions.
Speech Generation Synthesises speech in 36 languages/dialects; supports voice cloning and fine‑grained control of volume, speed, and emotion.
Long‑Form Reasoning The 256 k context enables document‑level analysis, multi‑turn dialogue, and complex tool use without truncation.

Performance Highlights

Qwen3.5‑Omni‑Plus (the largest instruction‑tuned variant) achieves state‑of‑the‑art results on 215 audio and audio‑visual benchmarks. Selected numbers (higher is better unless otherwise noted):

  • Audio‑Visual Understanding – surpasses Gemini‑3.1 Pro on general audio understanding, reasoning, recognition, translation, and dialogue; matches Gemini‑3.1 Pro on overall audio‑visual scores.
  • Audio‑Visual Captioning – produces controllable, detailed captions with automatic segmentation, timestamps, and character‑relationship descriptions.
  • Speech Synthesis Stability – measured by Word Error Rate (WER), Qwen3.5‑Omni‑Plus attains 12.62 % WER on a public 20‑language multilingual TTS set, outperforming ElevenLabs (13.08 %) and Gemini‑2.5 Pro (2.72 % on a different metric).
  • Multilingual Speech Recognition – achieves top‑ranked scores on 8 ASR benchmarks and 156 language‑specific speech‑to‑text tasks.
  • Vision & Text – on visual benchmarks (e.g., VQA, OCR, spatial reasoning) the model matches the performance of same‑size Qwen3.5 text‑only models.

New Interactive Features (Realtime API)

Qwen3.5‑Omni adds several capabilities that make it suitable for live assistants and conversational agents:

  1. Semantic Interruption – native turn‑taking intent detection prevents accidental back‑channeling and filters out background noise.
  2. WebSearch & Function Calling – the model can decide autonomously whether to invoke a web search or call external functions during a conversation.
  3. End‑to‑End Voice Control – users can issue spoken commands to adjust speaking volume, speed, or emotional tone on the fly.
  4. Voice Cloning – upload a short voice sample and the model will adopt that voice for subsequent replies.
  5. System Prompt Customisation – developers can modify the system prompt to change conversational style or persona without retraining.
  6. ARIA Alignment – ensures that streamed speech remains coherent even when text and speech token rates differ, dramatically improving naturalness.

Emerging Capability: Audio‑Visual Vibe Coding

During scaling experiments, Qwen3.5‑Omni demonstrated the ability to write code directly from audio‑visual instructions. The model can interpret a video demonstration (e.g., a UI interaction) and generate the corresponding source code, a phenomenon the authors term Audio‑Visual Vibe Coding. This capability is exposed via the Offline API.

Demo Highlights

  • Audio‑Visual Captioning – the model produces screenplay‑style, timestamped captions for a 3‑minute wildlife documentary, including character names, sound‑effect annotations, and scene transitions.
  • Multilingual Speech Interaction – a user speaks Mandarin, the model replies in a cloned Mandarin voice, then switches to English with a different cloned voice, all within the same session.
  • Tool Use – in a live chat, the model decides to perform a web search for the current date, retrieves the result, and incorporates it into the answer without explicit prompting.

Availability

Qwen3.5‑Omni is accessible through two endpoints:

  • Offline API – suitable for batch processing, large‑scale audio‑visual analysis, and the Audio‑Visual Vibe Coding use case.
  • Realtime API – optimized for low‑latency voice assistants, supporting the interactive features listed above.

Developers can obtain the models via the Qwen Chat interface, Hugging Face, or ModelScope (both offline and realtime demos are provided). The blog post includes example code for both offline and realtime invocations.

Implications

Qwen3.5‑Omni marks a concrete step toward native omnimodal AGI by unifying perception (vision, audio, video) and generation (text, speech, code) in a single architecture. Its large context window and multilingual speech capabilities broaden the range of applications—from multilingual media analysis and captioning to real‑time virtual assistants that can understand and respond to complex audio‑visual cues. The emergence of Audio‑Visual Vibe Coding suggests that future models may bridge the gap between demonstration and implementation, reducing the need for manual programming in many domains.

Citation

If you reference Qwen3.5‑Omni, please cite:

@misc{qwen35omniblog,
  title = {Qwen3.5-Omni: Scaling Up, Toward Native Omni-Modal AGI},
  url = {https://qwen.ai/blog?id=qwen3.5-omni},
  author = {Qwen Team},
  month = {March},
  year = {2026}
}

Sources