WorksApplications/SudachiDict

A lexicon for Sudachi

What it solves

It provides the essential lexicon (vocabulary data) required for the Sudachi Japanese tokenizer to function. Without a dictionary, a tokenizer cannot accurately split Japanese text into meaningful words or identify proper nouns.

How it works

SudachiDict provides pre-built binary dictionaries and source CSV files that the Sudachi tokenizer (and its Python/Rust implementations) loads to perform morphological analysis. It offers three levels of vocabulary coverage:

  • Small: Contains only UniDic vocabulary.
  • Core: The default version containing basic vocabulary.
  • Full: Includes additional miscellaneous proper nouns for broader coverage.

Who it’s for

Developers building Natural Language Processing (NLP) applications for Japanese text who are using the Sudachi tokenizer ecosystem.

Highlights

  • Multiple Tiers: Offers Small, Core, and Full versions to balance memory usage and vocabulary breadth.
  • Python Integration: Available as easy-to-install PyPI packages (sudachidict_core, etc.).
  • Versioned Formats: Maintains distinct binary formats (V0 and V1) to ensure compatibility with specific versions of the Sudachi engine.

Related

  • Project
    WorksApplications/Sudachi

    Sudachi is a Java‑based Japanese morphological analyzer that tokenises text, tags parts of speech, and normalises variants. It offers three segmentation modes, large UniDic‑based dictionaries (small, core, full), a plugin architecture, and support for custom user dictionaries. Use it via a command‑line jar, as a Maven library, or through ports in Python/Rust and an Elasticsearch plugin. Licensed under Apache 2.0.

  • Project
    WorksApplications/sudachi.rs

    A Rust implementation of the Sudachi Japanese morphological analyzer for high-performance tokenization and grammatical analysis of Japanese text.

  • Project
    lindera/lindera

    A high-performance morphological analysis library in Rust for tokenizing Japanese text, with bindings for Python, Node.js, Ruby, PHP, and WebAssembly.

  • Project
    ikawaha/kagome

    A pure Go Japanese morphological analyzer that tokenizes text and analyzes parts of speech with embedded dictionaries for easy deployment.

  • Project
    taishi-i/nagisa

    Nagisa is a Python library for Japanese word segmentation and part-of-speech tagging that uses recurrent neural networks to process text, offering simple tools for filtering, custom dictionaries, and model training.