pemistahl/lingua-rs

What it solves

Lingua is a language detection library designed to identify the language of a given text fragment. It specifically addresses the limitations of other libraries that struggle with very short text snippets (like single words or phrases) and lose accuracy as the number of supported languages increases.

How it works

Lingua combines a rule-based engine and a statistical Naive Bayes model. The rule-based engine first identifies the alphabet and searches for characters unique to specific languages to filter out impossible candidates. If a unique language is found, it skips the statistical model entirely. Otherwise, it uses a probabilistic n-gram model with n-gram sizes ranging from 1 to 5, which provides higher accuracy for short texts compared to the standard trigram models used by most other libraries. To maintain efficiency, language models are stored as finite-state transducers (FSTs), allowing them to be searched on disk without being fully loaded into memory.

Who it’s for

It is intended for developers building natural language processing (NLP) applications, such as text classifiers, spell checkers, or automated email routing systems, who need a lightweight, offline, and highly accurate language detection tool without the overhead of a full machine learning framework.

Highlights

• High Accuracy for Short Text: Optimized for single words and short phrases.
• Support for 75 Languages: Focuses on quality over quantity for its supported set.
• Hybrid Approach: Uses both rule-based filtering and statistical n-gram models (1-5 grams).
• Low Resource Usage: Uses FSTs to keep memory consumption low, making it suitable for low-resource environments.
• Offline Capability: Works entirely offline with no external API calls.
• Multi-language Support: Available as a Rust library and a Python extension module.