Qwen 3.8 Omni-Flash release: 1M-token omnimodal model with agentic audio‑visual capabilities
TL;DR
Qwen3.8‑Omni‑Flash is a new 1 M‑token omnimodal model that delivers agentic audio‑visual reasoning, long‑form video understanding, and real‑time interaction while cutting audio input costs by >98% and audio‑visual input costs by >93% compared with its predecessor.
Core Advances
Answer‑first: Qwen3.8‑Omni‑Flash moves omnimodal AI from passive perception to autonomous task planning, tool use, and content creation across text, image, audio, and video.
- 1 M‑token context window with text performance on par with a text‑only model of the same size.
- Benchmark gains: average score +25% over Qwen3.5‑Omni‑Plus across 29 evaluations; +36.5 points on WildClawBench‑MM, +22.3 on AgenticVBench, and a strong 69.6 on UniClawBench.
- Audio‑visual pricing: hourly audio input price drops from 1.5 CNY to 0.15 CNY; audio‑visual price drops from 9.0 CNY to 0.47 CNY (≈93‑98% reduction).
- Performance parity with Gemini 3.8 Flash: audio‑visual scores are close to Gemini 3.8 Flash and overall audio performance exceeds it.
Long‑Form Audio‑Visual Understanding
Controllable Captioning
Users can dictate subject, time range, detail level, and output format, enabling a single model to produce an overview, segment highlights, or deep analyses of character actions, lighting, and sound.
"Users can freely specify the subject, time range, level of detail, and output format. For the same video, the model can provide an overview, locate key segments, or analyze character actions, camera shots, lighting, and sound in depth, producing structured results as needed." – Qwen blog
Agentic Long‑Form QA
The model starts from a question, decides which portions of a multi‑hour video to inspect, and gathers evidence in a coarse‑to‑fine loop. On OmniVideoBench, accuracy improves from 63.4 % (static) to 67.8 % (agentic) while token usage drops 45.7 % (145,736 → 79,117 tokens).
| Mode | Accuracy ↑ | Tokens per query ↓ |
|---|---|---|
| Static | 63.4 | 145,736 |
| Agentic | 67.8 | 79,117 |
Multi‑Speaker Meeting Intelligence
Qwen3.8‑Omni‑Flash jointly diarizes speakers across audio and video, supports up to one hour of input, and can generate minutes, action items, and risk analyses. It can subsequently trigger tools (e.g., email, code generation) to act on meeting outcomes.
Production‑Grade Audio‑Visual Workflows
Music‑to‑MV
The model parses rhythm, mood, and vocal cues, creates scene‑by‑scene storyboards, and timestamps lyrics for precise subtitle alignment.
Short‑Drama Localization
A single natural‑language request triggers speaker‑aware transcription, translation, voice cloning, audio remixing, and final editing, eliminating the need for multiple platform switches.
Long‑Form Film Commentary
For full‑length movies, the model extracts plot points, plans commentary structure, synthesizes voice‑over, mixes music, and renders a finished commentary video with seamless interleaving of original dialogue.
Model‑in‑the‑Loop Optimization
Qwen3.8‑Omni‑Flash was tasked with improving the Sichuan‑dialect ASR of the 3 B‑parameter Qwen2.5‑Omni model. Over four experiment cycles it generated 3,413 targeted training examples, achieving a 40.7 % relative reduction in character error rate (25.79 % → 15.30 %).
Information Compression Tools
Video2Note
Open‑source Video2Note plugin converts hours of tutorial video into structured PDF notes with timestamps, key frames, and step‑by‑step instructions.
Omni Skill Creator
Omni‑Skill‑Creator extracts SOPs or tool‑usage patterns from demonstration videos, producing reusable Skill.md files that can be invoked by agents for repeatable automation.
Real‑Time Interaction (Qwen‑Omni‑Flash‑Realtime)
The realtime API streams audio‑visual input at 1 fps (720p) and returns text, audio, or tool‑calling responses within ~600 ms to first token and ~1 s to first audio packet. Use cases include:
- Speaking practice – the model aligns accented speech with standard pronunciation and provides corrective feedback on the fly.
- Spatial audio perception – it localizes sound sources in 3‑D space, enabling commands like “go see that sound.”
- Dynamic knowledge injection – Skills can load brand‑specific language, business rules, or domain knowledge at runtime, allowing the same model to act as a customer‑service agent, a robot navigator, or a personal tutor.
Benchmark Highlights
Omni Agentic Benchmarks
| Metric | Qwen3.8‑Omni‑Flash | Qwen3.5‑Omni‑Plus | Gemini 3.8 Flash |
|---|---|---|---|
| WildClawBench‑MM (tool use) | 71.0 | 34.5 | 58.9 |
| UniClawBench (tool use) | 69.6 | 67.1 | 69.0 |
| AgenticVBench (tool use) | 36.8 | 14.5 | 45.0 |
| OmniGAIA (web search) | 74.0 | 57.2 | 78.6 |
General Audio‑Visual Understanding
| Benchmark | Qwen3.8‑Omni‑Flash | Gemini 3.8 Flash |
|---|---|---|
| DailyOmni | 85.1 (tied) | 84.0 |
| WorldSense | 68.5 | 69.6 |
| AVUT | 86.6 | 88.0 |
| JoinAVBench | 75.9 | 70.4 |
| OmniVideoBench (reasoning) | 63.4 (static) / 67.8 (agentic) | 65.2 |
| Video‑MME‑v2 | 65.0 (static) / 71.3 (agentic) | 71.0 |
| LVOmniBench (long video) | 63.3 (static) / 73.6 (agentic) | 70.7 |
Speech Recognition (multi‑speaker)
| Dataset | Qwen3.8‑Omni‑Flash | Prior Qwen | Gemini 3.8 Flash |
|---|---|---|---|
| AliMeeting (DER / cpWER) | 3.4 / 3.35 | 17.2 / 89.6 | 88.1 / 89.6 |
| AISHELL‑4 | 2.8 / 2.8 | 11.2 / 66.4 | 100 / 100 |
| MLC‑SLM (en) | 4.0 / 4.0 | 14.2 / 60.8 | 68.6 / 63.9 |
Throughput & Latency (Realtime API)
| Input | Tokens/s (TPS) | Time‑to‑First‑Token (ms) | Time‑to‑First‑Audio‑Packet (ms) | Audio RTF |
|---|---|---|---|---|
| Audio 6 s | 84.87 | 591 | 978 | 0.154 |
| Audio‑Visual 12 s | 84.33 | 912 | 1,269 | 0.153 |
Language Coverage
- Speech recognition: 74 languages + 39 Chinese dialects (e.g., Sichuanese, Cantonese, Shanghainese).
- Speech generation: 29 languages + 7 Chinese dialects.
Getting Started
API (Chat‑Completions compatible)
from openai import OpenAI
import os
client = OpenAI(
api_key=os.getenv("DASHSCOPE_API_KEY"),
base_url=os.getenv("DASHSCOPE_BASE_URL", "https://dashscope-intl.aliyuncs.com/compatible-mode/v1"),
)
messages = [{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}},
{"type": "input_audio", "input_audio": {"data": "https://example.com/audio.wav", "format": "wav"}},
{"type": "text", "text": "Describe the image and transcribe the audio."},
],
}]
completion = client.chat.completions.create(
model="qwen3.8-omni-flash",
messages=messages,
reasoning_effort="xhigh",
stream=True,
)
for chunk in completion:
if hasattr(chunk.choices[0].delta, "reasoning_content"):
print(chunk.choices[0].delta.reasoning_content, end="")
if hasattr(chunk.choices[0].delta, "content"):
print("\nAnswer:", chunk.choices[0].delta.content)
reasoning_effortoptions:xhigh(default),medium,low.preserve_thinkingis enabled by default for full reasoning trace.
Installing Qwen‑MM‑Plugins (agent harness)
curl -fsSL https://raw.githubusercontent.com/QwenLM/Qwen-MM-Plugins/main/install.sh | bash
# Choose: Install → <your harness> → omni‑related plugins
Key plugins:
| Plugin | Capability |
|---|---|
core |
Reads images, video frames, PDFs, code, 3D files |
api |
OCR, audio‑visual transcription, speaker diarization |
omni‑chatcut |
MV creation, film commentary, video translation |
omni‑video2note |
Auto‑generates PDF notes from tutorials |
omni‑skill‑creator |
Extracts reusable SOPs from demos |
omni‑memory |
Builds long‑term memory of people, sounds, events |
Real‑Time API (WebSocket example)
import os, base64, time, pyaudio, cv2
from dashscope.audio.qwen_omni import (
MultiModality, AudioFormat, OmniRealtimeCallback, OmniRealtimeConversation,
)
url = "wss://dashscope-intl.aliyuncs.com/api-ws/v1/realtime"
model = "qwen3.8-omni-flash-realtime"
class SimpleCallback(OmniRealtimeCallback):
def on_event(self, response):
if response['type'] == 'response.audio.delta':
self.out.write(base64.b64decode(response['delta']))
elif response['type'].endswith('transcription.delta'):
print('\r[User]', response.get('text','')+response.get('stash',''), end='')
elif response['type'].endswith('transcription.completed'):
print('\r[User]', response['transcript'])
elif response['type'] == 'response.audio_transcript.done':
print('[LLM]', response['transcript'])
# ... (initialize audio/video streams as in the blog) ...
The open‑source Qwen‑Live‑Harness (npm i -g qwen-live-harness) provides a ready‑made CLI for continuous audio‑visual conversations, tool delegation, and memory management.
Community Reaction (Hacker News Highlights)
"If the performances are comparable, and there is no evidence it's not. in/out ($) Gemini : 1.5 / 9.0 | Qwen 3.8: 0.15 / 0.47 That is a massive cost reduction." – @ache
*"audio‑visual performance close to Gemini 3.8 Flash and overall audio performance that exceeds Gemini 3.8 Flash" – many commenters note the claim is impressive given Gemini’s strong multi‑language audio support.
"Looks like the harness repo is already removed?" – @lxe points out a broken link, suggesting the live‑harness repo may have been taken down after the announcement.
"Qwen offers many model sizes, making it easy to experiment with tiny LLMs." – @tolugenius highlights the breadth of the Qwen family.
Bottom Line
Qwen3.8‑Omni‑Flash represents a significant step toward truly agentic, omnimodal AI: it can ingest up to 1 M tokens of mixed media, reason efficiently over long audio‑visual streams, and execute end‑to‑end production pipelines—all at a fraction of the cost of competing offerings. The open‑source plugin ecosystem and real‑time harness further lower the barrier for developers to build sophisticated multimodal agents.
Sources
Related
- Dispatch
- Dispatch
- Dispatch
- Dispatch
- Dispatch