QwenAudio/SenseVoice

Open-source SenseVoiceSmall model for Mandarin, Cantonese, English, Japanese, and Korean ASR, language ID, emotion recognition, and audio event detection.

What it solves

SenseVoice is a speech foundation model designed to provide high-accuracy multilingual speech recognition and understanding. It addresses the need for a single model that can simultaneously handle automatic speech recognition (ASR), spoken language identification (LID), speech emotion recognition (SER), and audio event detection (AED).

How it works

SenseVoice uses a non-autoregressive end-to-end framework, which allows for low-latency inference. The released SenseVoiceSmall checkpoint supports Mandarin, Cantonese, English, Japanese, and Korean. It can be deployed as a Python package via FunASR, exported to ONNX or Libtorch, or run as a self-contained binary using llama.cpp/GGUF for CPU/edge environments. For long audio files, it employs a bounded-memory windowing approach or a VAD (Voice Activity Detection) pipeline to prevent memory overflows.

Who it’s for

This project is for developers and researchers needing efficient, multilingual speech-to-text systems that also capture emotional context and environmental audio events, particularly those working with Chinese and Cantonese languages.

Highlights

  • Multitask Capability: Performs ASR, language identification, emotion recognition, and audio event detection (e.g., laughter, coughing, applause) in one model.
  • High Efficiency: The non-autoregressive architecture makes it significantly faster than Whisper-Small and Whisper-Large at similar parameter counts.
  • Rich Transcription: Surpasses many existing models in speech emotion recognition and provides sound event tags.
  • Flexible Deployment: Supports various runtimes including FastAPI, Docker, and a C++ binary for no-Python environments.

Related

  • Project
    modelscope/FunASR

    FunASR is an industrial-grade speech recognition toolkit from Alibaba's ModelScope team. It provides a modular Python API, CLI, and server for offline/streaming ASR, plus VAD, punctuation, speaker diarization, emotion recognition, and edge deployment via llama.cpp. It includes multiple pre-trained models (e.g., Fun-ASR-Nano, SenseVoiceSmall) and supports GPU, CPU, and edge devices.

  • Project
    QwenAudio/Fun-ASR

    A family of high-precision end-to-end speech recognition models optimized for noisy environments, far-field audio, and extensive Chinese dialect support.

  • Project
    QwenAudio/CosyVoice

    CosyVoice is an advanced, LLM-based text-to-speech system designed for high-quality, zero-shot multilingual speech synthesis and voice cloning.

  • Project
    xzf-thu/Mega-ASR

    Mega-ASR is a foundation ASR model designed for robust speech recognition in challenging real-world environments by scaling up acoustic simulations of noise, echo, and distortion.

  • Project
    istupakov/onnx-asr

    A lightweight Python package for Automatic Speech Recognition using ONNX models, eliminating the need for PyTorch or Transformers for fast deployment on edge and server hardware.