netease-youdao/Confucius4-R2T2

What it solves

Confucius4-R2T2 addresses the challenge of creating real-time Automatic Speech Recognition (ASR) that is both low-latency and high-accuracy. Specifically, it solves the problem of "visual flickering" and disruptive text revisions common in many streaming ASR systems by using an append-only output mode, where emitted text is committed permanently and never revised.

How it works

Built upon the Qwen3-ASR model, R2T2 utilizes a Longest Stable Prefix (LSP) learning paradigm. This allows the model to dynamically determine when a segment of text (a stable prefix) can be safely emitted without needing further audio context. To achieve this, it was trained using specialized data construction techniques, including stable-prefix data, forced time-alignment data, and token-level audio segmentation. It supports configurable decoding chunks ranging from 80 ms to 2 s to allow users to balance the trade-off between latency and accuracy.

Who it’s for

This project is designed for developers and researchers building applications that require instant, stable text output, such as:

• Real-time live captioning and subtitling.
• Downstream NLP pipelines and LLM agents.
• Simultaneous speech translation systems.

Highlights

• True Streaming Output: Emits text in an append-only mode, ensuring previously emitted words are not revised.
• Low Latency: Achieves average latency between 200 and 600 milliseconds.
• High Accuracy: Maintains recognition quality close to offline ASR performance.
• Flexible Configuration: Supports decoding chunks from 80 ms to 2 s.
• High Throughput: Powered by a vLLM backend for efficient inference.
• Multilingual: Optimized for Chinese and English, with support for various other languages.