WorksApplications/SudachiDict

A lexicon for Sudachi

What it solves

It provides a comprehensive lexicon for the Sudachi Japanese tokenizer, enabling accurate word segmentation and morphological analysis of Japanese text.

How it works

The project offers pre-built dictionaries in various sizes to balance memory usage and coverage. It integrates with SudachiPy (the Python version of Sudachi) as installable Python packages, allowing users to specify the dictionary edition they need for their NLP pipeline.

Who it’s for

Developers and researchers working with Japanese Natural Language Processing (NLP) who need a reliable lexicon for tokenization.

Highlights

  • Available in three sizes: Small (UniDic vocabulary), Core (basic vocabulary), and Full (includes miscellaneous proper nouns).
  • Easy installation via pip for Python users.
  • Based on UniDic and parts of NEologd.
  • Licensed under Apache License 2.0.

Related

  • Project
    WorksApplications/Sudachi

    Sudachi is a Java‑based Japanese morphological analyzer that tokenises text, tags parts of speech, and normalises variants. It offers three segmentation modes, large UniDic‑based dictionaries (small, core, full), a plugin architecture, and support for custom user dictionaries. Use it via a command‑line jar, as a Maven library, or through ports in Python/Rust and an Elasticsearch plugin. Licensed under Apache 2.0.

  • Project
    WorksApplications/sudachi.rs

    A Rust implementation of the Sudachi Japanese morphological analyzer for high-performance tokenization and grammatical analysis of Japanese text.

  • Project
    lindera/lindera

    A high-performance morphological analysis library in Rust for Japanese text tokenization, providing bindings for Python, Node.js, and other languages.

  • Dispatch
    Context-Aware Furigana: Solving the Ambiguity of Japanese Readings

    Explore how EZFurigana leverages Sudachi and ModernBERT to solve the context-dependent reading challenges of the Japanese language.

  • Project
    ikawaha/kagome

    A pure Go Japanese morphological analyzer that tokenizes text and analyzes parts of speech with embedded dictionaries for easy deployment.