lindera/lindera

A multilingual morphological analysis library.

What it solves

Lindera is a morphological analysis library designed to tokenize and analyze Japanese text. It provides a way to break down Japanese sentences into their constituent parts (morphemes) and assign linguistic properties to them, which is a critical first step for many natural language processing tasks.

How it works

Written in Rust for performance, the library uses dictionaries (such as IPADIC) to perform morphological analysis. It offers concise APIs for integration into Rust applications, and also provides bindings for Python and WebAssembly to make the analysis tools available across different environments.

Who it’s for

Developers building applications that require Japanese text processing, such as search engines, sentiment analysis tools, or other NLP pipeline components, who need a fast and easy-to-install library.

Highlights

  • High performance: Tokenizes Japanese text at approximately 10-20 MB/s single-threaded.
  • Multi-language support: Available as a Rust library, with Python and WebAssembly bindings.
  • Easy installation: Provides a pre-built dictionary download system and a CLI for automatic dictionary installation.

Related

  • Project
    huggingface/tokenizers

    A high-performance Rust-based library for training and implementing the most widely used text tokenization algorithms for AI models.

  • Project
    ikawaha/kagome

    A pure Go Japanese morphological analyzer that tokenizes text and analyzes parts of speech with embedded dictionaries for easy deployment.

  • Project
    zibo-chen/rust-paddle-ocr

    A lightweight Rust OCR library based on PaddleOCR models and the MNN inference runtime, providing text detection and recognition across multiple languages.

  • Project
    pemistahl/lingua-py

    A high-accuracy language detection library that identifies the languages of text snippets, specifically optimized for short phrases and single words.

  • Project
    daac-tools/vibrato

    A fast Rust-based tokenizer and morphological analyzer that optimizes the Viterbi algorithm for high-speed tokenization, especially for large dictionaries.