pemistahl/lingua-py

What it solves

Lingua is a language detection library designed to accurately identify the language of a given text, even for very short snippets like single words or phrases. It addresses the limitations of other libraries that often struggle with short text or lose accuracy as the number of supported languages increases.

How it works

Lingua combines a rule-based engine and a statistical Naive Bayes model. The rule-based engine first identifies the alphabet and searches for unique characters to filter out impossible languages. Then, it uses a probabilistic n-gram model (supporting n-gram sizes from 1 to 5) to determine the most likely language. To maintain high performance and low memory usage, the language models are stored as finite-state transducers (FSTs), allowing them to be searched on disk without being fully loaded into memory. The core implementation is written in Rust with Python bindings.

Who it’s for

It is intended for developers building natural language processing (NLP) applications, such as text classification, spell checkers, or automated email routing, who need a lightweight, offline, and accurate language detection tool without the overhead of large ML frameworks.

Highlights

• High accuracy on short text: Uses n-grams of sizes 1-5 to handle single words and short phrases effectively.
• Offline capability: Works completely offline without requiring external APIs.
• Broad language support: Supports 75 different languages.
• Efficient resource usage: Utilizes Rust-based bindings and FSTs for a small memory footprint and fast execution.
• Confidence values: Provides probability scores for each candidate language.
• Thread-safe: The library is designed to be thread-safe for use in multi-threaded environments.