antirez/qwen-asr
C inference for Qwen3-ASR 0.6b and 1.7b transcriptions models
What it solves
This project provides a high-performance, pure C implementation of the inference pipeline for Qwen3-ASR speech-to-text models. It allows users to run powerful ASR models (0.6B and 1.7B parameters) with minimal dependencies, enabling efficient transcription on CPUs, NVIDIA GPUs (via CUDA), and AMD GPUs (via ROCm).
How it works
The implementation uses a C-based pipeline that handles audio preprocessing, tokenization, and model inference. It supports three primary modes of operation:
- Normal Mode: Processes audio in one shot (offline) or in segments to handle larger files.
- Streaming Mode: Processes audio in 2-second chunks with a sliding window and prefix rollback to maintain context and stability for live audio.
- Segmented Mode: Splits long audio into segments based on silence boundaries to manage memory and compute costs.
To optimize performance, it uses BLAS for CPU acceleration and native kernels for GPU backends. It also employs memory-mapped weights for CPU inference to reduce memory overhead.
Who it’s for
- Developers looking for a lightweight, dependency-free C library for integrating speech-to-text into applications.
- System Administrators needing to run ASR on remote Linux servers without heavy Python runtimes.
- Users wanting to run Qwen3-ASR models on modest hardware, including low-end Intel/AMD processors or specific AMD/NVIDIA GPUs.
Highlights
- Minimal Dependencies: CPU build requires only the C standard library and a BLAS implementation.
- GPU Acceleration: Native support for NVIDIA CUDA and AMD ROCm backends.
- Flexible Input: Supports WAV files and audio piped via stdin (e.g., from ffmpeg).
- Real-time Diagnostics: A "monitor mode" that uses Unicode symbols to visualize the pipeline's progress on stderr.
- Language & Prompt Control: Ability to force specific target languages or provide system prompts to bias spelling and technical terms.
- Streaming Capabilities: Support for indefinite streaming with a sliding window to bound memory and compute.
Related
- Project
- Project
- Project
- Project