WorksApplications/Sudachi

Sudachi – Japanese Morphological Analyzer

Sudachi is a Japanese tokeniser / morphological analyzer written in Java. It takes raw Japanese text and outputs a sequence of morphemes together with part‑of‑speech tags, dictionary (base) forms, readings, and optional extra information. The project focuses on three core tasks:

1. Segmentation – breaking a sentence into words (or sub‑words) using three selectable modes (A = shortest units, B = middle units, C = named‑entity‑style units).
2. POS tagging – assigning detailed part‑of‑speech labels.
3. Normalization – unifying spelling variants, script differences, misspellings, and contracted forms.

Why Sudachi?

• Multiple‑length segmentation lets you switch between fine‑grained (A), medium (B), or entity‑oriented (C) splits, which is useful for search, NLP pipelines, or downstream models.
• Large lexicon built on UniDic plus the NEologd extension, with three pre‑packaged dictionaries (small, core, full) to balance memory vs coverage.
• Plugin architecture enables custom input normalisation, OOV handling, word‑connection rules, and path modifications without changing the core code.
• User dictionaries can be added at runtime, allowing domain‑specific vocabularies.
• Integration: provides a command‑line tool, a Maven‑compatible Java library, and companion plugins for Elasticsearch, plus ports in Python and Rust.

Getting Started

1. Installation – Follow the tutorial in docs/tutorial.md to download a pre‑built dictionary and run the JAR.
2. Command‑line usage – Example:

   echo 東京都へ行く | java -jar sudachi.jar -m A   # short‑unit split
   

   Options include -t (space‑separated output), -a (extra fields), --print-reading, and dictionary selection via --systemDict.
3. Library usage – Add the Maven dependency:

   <dependency>
     <groupId>com.worksap.nlp</groupId>
     <artifactId>sudachi</artifactId>
     <version>0.5.3</version>
   </dependency>
   

   Then use the Java API (see Javadoc) to tokenise programmatically.

Plugins & Extensibility

Sudachi ships with several ready‑made plugins (character normalisation, long‑sound normalisation, removal of yomigana, one‑character OOV handling, MeCab‑compatible OOVs, inhibition rules, numeric normalisation, etc.). New plugins can be written to hook into any processing stage: input text, unknown‑word handling, word‑connection costs, or output path adjustments.

Comparison to Other Japanese Tokenisers

Feature	Sudachi	MeCab	Kuromoji
Multiple segmentation lengths	✅	❌	Limited
Built‑in normalisation (script, misspelling)	✅	❌	Limited
User‑dictionary stacking	✅	✅	❌
Memory efficiency (shared mmap)	Good	Poor	Good
Accuracy	Good	Good	Good
Speed	Good	Excellent	Good

Ecosystem

• Elasticsearch plugin – WorksApplications/elasticsearch-sudachi for full‑text search integration.
• Python / Rust ports – WorksApplications/sudachi.rs provides native bindings for other language ecosystems.
• Community – Slack workspace for developers and users.

License & Citation

Sudachi is released under the Apache License 2.0. If you use it in research, cite the LREC 2018 paper “Sudachi: a Japanese Tokenizer for Business” (BibTeX provided in the README).

---

In a nutshell: Sudachi is a mature, extensible Japanese morphological analyser that offers flexible tokenisation modes, rich dictionary resources, and a plugin system, making it suitable for search engines, NLP pipelines, and any application needing accurate Japanese word segmentation and POS tagging.