OpenMOSS/MOSS-Audio

What it solves

MOSS-Audio provides a unified model for complex real-world audio understanding. It moves beyond simple transcription to perceive acoustic cues, recognize speakers and emotions, interpret environmental sounds, and perform multi-step reasoning over temporal audio contexts.

How it works

The system uses a modular architecture consisting of three parts:

1. MOSS-Audio-Encoder: A dedicated encoder trained from scratch that converts raw audio into temporal representations at 12.5 Hz.
2. Modality Adapter: Projects these representations into the embedding space of a Large Language Model (LLM).
3. LLM Backbone: Uses Qwen3 (4B or 8B) to generate auto-regressive text based on the audio features.

To improve performance, it employs DeepStack Cross-Layer Feature Injection, which feeds features from multiple encoder layers (not just the top layer) into the LLM to preserve low-level acoustic details like timbre and rhythm. It also uses time-marker insertion during pretraining to help the model track "what happened when," enabling accurate timestamping and event localization.

Who it’s for

Developers and researchers needing a high-performance, open-source audio model for tasks like speech-to-text with timestamps, audio captioning, environmental sound analysis, and complex audio-based question answering.

Highlights

• Unified Capabilities: Supports speech understanding, music analysis, environmental sound detection, and audio captioning in one model.
• Reasoning Variants: Offers both "Instruct" models for direct following and "Thinking" models for chain-of-thought reasoning.
• High Temporal Accuracy: Dramatically outperforms other open-source and some closed-source models in timestamp ASR accuracy.
• Robust ASR: Strong performance across diverse scenarios, including code-switching, dialects, and non-speech vocalizations.