Qwen2-Audio Release Notes

Qwen has released Qwen2-Audio, an audio-language model that accepts audio and text inputs to generate text outputs. This model enables direct voice interaction without the need for separate Automatic Speech Recognition (ASR) modules and provides advanced audio analysis capabilities.

Key Capabilities

Qwen2-Audio provides three primary functional features:

  • Voice Chat: Users can provide instructions directly via audio, allowing the model to respond to spoken commands without an intermediate ASR step.
  • Audio Analysis: The model can analyze various audio types—including speech, music, and general sound—based on text instructions.
  • Multilingual Support: The model supports more than eight languages and dialects, including English, Chinese, Cantonese, French, Italian, Spanish, German, and Japanese.

Performance and Benchmarks

Qwen2-Audio significantly outperforms both the original Qwen-Audio and other state-of-the-art (SOTA) models across a series of benchmark datasets. These benchmarks include:

  • LibriSpeech
  • Common Voice 15
  • Fleurs
  • Aishell2
  • CoVoST2
  • Meld
  • Vocalsound
  • AIR-Benchmark

Technical Architecture

The model is built using a foundation consisting of a Qwen language model and an audio encoder. The training process follows a three-step sequence:

  1. Multi-task Pretraining: Used for audio-language alignment.
  2. Supervised Finetuning (SFT): Used to grasp downstream task capabilities.
  3. Direct Preference Optimization (DPO): Used to align the model with human preferences.

Implementation and Usage

Qwen2-Audio is officially supported by the Hugging Face transformers library. Users can implement the model in two primary modes:

  • Voice Chat Mode: Input consists solely of audio; the model interprets instructions contained within the audio.
  • Audio Analysis Mode: Input consists of audio paired with specific text instructions.

Open-weight versions, including Qwen2-Audio-7B and Qwen2-Audio-7B-Instruct, are available on Hugging Face and ModelScope.

Future Roadmap

Qwen plans to expand the capabilities of Qwen2-Audio in the following areas:

  • Dataset Expansion: Training on larger pretraining datasets to support audio files longer than 30 seconds.
  • Model Scaling: Developing larger versions of the model to further explore the scaling laws of audio language models.

Sources