WorksApplications/Sudachi

Sudachi – Japanese Morphological Analyzer

Sudachi is an open‑source Japanese tokenizer and morphological analyzer written in Java. It splits Japanese text into morphemes, assigns part‑of‑speech tags, and normalizes the output (e.g., handling variant spellings, script conversions, and common misspellings). The project is maintained by Works Applications Co., Ltd. and released under the Apache 2.0 license.

Core capabilities

• Multiple‑length segmentation – three built‑in modes (A, B, C) let you choose the granularity of the split, from the shortest UniDic units (A) to named‑entity‑style chunks (C), with a middle‑ground (B).
• Large lexicon – dictionaries are built on UniDic and the NEologd extensions, offering a core set plus optional small and full dictionaries that add more proper nouns.
• Plugin system – you can attach plugins to modify input normalization, handle out‑of‑vocabulary (OOV) words, adjust word‑connection costs, or post‑process the path (e.g., person‑name correction). Several ready‑made plugins are shipped (character case/width normalization, long‑sound‑mark handling, yomigana removal, etc.).
• User dictionaries – custom dictionaries can be added at runtime without replacing the system dictionary.
• Command‑line tool – a java -jar sudachi-XX.jar executable accepts a variety of flags for dictionary selection, output format, segmentation mode, and debugging.
• Java library – the same engine can be used programmatically via Maven (com.worksap.nlp:sudachi). Javadoc provides the full API.

Typical usage (CLI)

# basic tokenisation (default core dictionary)
echo 東京都へ行く | java -jar sudachi-0.8.2.jar

# show dictionary form, reading, and extra metadata
echo 東京都へ行く | java -jar sudachi-0.8.2.jar -a

# select a specific dictionary
java -jar sudachi-0.8.2.jar -s '{"systemDict":"system_small.dic"}'

# choose segmentation mode
java -jar sudachi-0.8.2.jar -m C   # named‑entity style

Integration points

• Maven dependency – add the sudachi artifact to a Java project and call the API for on‑the‑fly analysis.
• Elasticsearch plugin – a separate repository (elasticsearch-sudachi) lets you plug Sudachi into Elasticsearch for Japanese full‑text search.
• Other language ports – the same algorithm is re‑implemented in Python and Rust (sudachi.rs).

Development status

• The 0.8.x series is marked unstable; breaking changes may appear between patch releases, so pinning an exact version is recommended.
• v0.8.2 introduced a new binary dictionary format (V1). Upgrading requires rebuilding any user dictionaries against the matching system dictionary.
• Future work listed by the authors includes speed improvements, more plugins, richer split information, and integration with a synonym dictionary.

License

Sudachi is released under the Apache License 2.0, allowing free use, modification, and distribution.

---

All details above are taken directly from the repository’s README.