daac-tools/vibrato

What it solves

Vibrato is a high-performance morphological analyzer and tokenizer. It addresses the need for faster tokenization, particularly for languages with large dictionaries (matrices), where traditional tools like MeCab may be slower due to cache inefficiency.

How it works

It is a Rust-based reimplementation of the MeCab algorithm using the Viterbi algorithm for tokenization. It optimizes performance by using cache-efficient ID mappings, which significantly reduces tokenization time, especially when dealing with large-scale language resources.

Who it’s for

Developers and researchers working with Natural Language Processing (NLP) who require high-speed tokenization and morphological analysis, especially for languages like Japanese, and those who need compatibility with MeCab-style dictionaries and output formats.

Highlights

• High Speed: Optimized for CPU cache efficiency to outperform MeCab and its reimplementations.
• MeCab Compatibility: Supports identical output results and options (such as ignoring whitespace) to ensure a seamless transition from MeCab.
• Customizable: Supports user dictionaries in CSV format and the ability to train parameters/costs from corpora.
• Multi-platform: Implemented in Rust with a Python wrapper and a Wasm demo available.