lindera/lindera

A multilingual morphological analysis library.

What it solves

Lindera is a morphological analysis library designed to tokenize Japanese text, which is a language that does not use spaces between words. It provides a concise API for developers to break down Japanese text into its constituent parts for further processing.

How it works

Written in Rust for performance, the library uses dictionaries (such as IPADIC) to analyze text. It offers a pre-built dictionary system where users can download archives and specify the path to the load them. The core logic is implemented in Rust, but it provides bindings for multiple languages including Python, Node.js, Ruby, PHP, and WebAssembly for use in browsers.

Who it’s for

Developers building applications that need to process Japanese text, specifically those requiring efficient tokenization and morphological analysis across different programming environments.

Highlights

  • High performance: Tokenizes Japanese text at roughly 10-20 MB/s single-threaded.
  • Multi-language support: Official bindings for Python, Node.js, Ruby, PHP, and WebAssembly.
  • Easy installation: Provides a CLI tool for automatic dictionary downloads.
  • Rust-native: Built in Rust for safety and safety and speed.

Related

  • Project
    WorksApplications/sudachi.rs

    A Rust implementation of the Sudachi Japanese morphological analyzer for high-performance tokenization and grammatical analysis of Japanese text.

  • Project
    ikawaha/kagome

    A pure Go Japanese morphological analyzer that tokenizes text and analyzes parts of speech with embedded dictionaries for easy deployment.

  • Project
    polm/cutlet

    A tool for converting Japanese text to romaji with support for multiple romanization systems and URL slug generation.

  • Project
    megagonlabs/ginza

    GiNZA is a Python library for Japanese NLP that provides fast, accurate dependency parsing, POS tagging, lemmatization, and named‑entity recognition. It offers a standard model and transformer‑based models, supports GPU acceleration, includes CLI tools (`ginza`, `ginzame`), and integrates with spaCy for easy Python use.

  • Project
    WorksApplications/Sudachi

    Sudachi is a Java‑based Japanese morphological analyzer that tokenises text, tags parts of speech, and normalises variants. It offers three segmentation modes, large UniDic‑based dictionaries (small, core, full), a plugin architecture, and support for custom user dictionaries. Use it via a command‑line jar, as a Maven library, or through ports in Python/Rust and an Elasticsearch plugin. Licensed under Apache 2.0.