quickwit-oss/whichlang

What it solves

Whichlang provides a high-performance, lightweight solution for detecting the language of a given text string. It was developed to meet the high-throughput requirements of log and tracing data processing, where standard language detection libraries were either too slow or insufficiently accurate.

How it works

The library uses a multiclass logistic regression model. It extracts features from the input text using 2, 3, and 4-character n-grams, as well as codepoint analysis. These features are projected into a 4,096-dimensional space using the hashing trick. The resulting model is highly optimized for speed, achieving throughputs exceeding 100 MB/s.

Who it’s for

It is designed for developers working in Rust who need to integrate fast, reliable language identification into data-intensive applications, such as search engines, log processors, or real-time analytics pipelines.

Highlights

• Extremely fast: Achieves throughput over 100 MB/s, significantly faster than existing alternatives.
• High accuracy: Provides ~97% average accuracy across 16 supported languages.
• Zero dependencies: Lightweight implementation suitable for performance-critical environments.
• Supports 16 languages including Arabic, Mandarin, English, French, German, Hindi, Italian, Japanese, Korean, Portuguese, Russian, Spanish, Swedish, Turkish, and Vietnamese.