wenet-e2e/WeTextProcessing

What it solves

WeTextProcessing provides a production-ready toolkit for text normalization (TN) and inverse text normalization (ITN). It handles the conversion of written text into spoken forms (e.g., converting "2.5" to "two point five") and vice versa, which is essential for speech-to-text and text-to-speech systems.

How it works

The toolkit uses Finite State Transducers (FST) to process text. It supports multiple languages, including Chinese and English, and allows users to define custom rules to handle specific linguistic patterns. To optimize performance, FST graphs are automatically cached, though they can be rebuilt manually if rules are changed.

Who it’s for

This tool is designed for developers building speech-related AI applications, such as ASR (Automatic Speech Recognition) and TTS (Text-to-Speech) pipelines, who need reliable, scalable text processing for production environments.

Highlights

• Bidirectional Processing: Supports both Text Normalization (TN) and Inverse Text Normalization (ITN).
• Multi-language Support: Includes built-in pipelines for Chinese and English.
• FST-based Architecture: Utilizes OpenFst and Pynini for efficient rule-based transformations.
• Production Ready: Features a C++ runtime for deployment and automatic caching of processing graphs.
• Detailed Mapping: Ability to retrieve changed spans and token types between input and output text.