WorksApplications/sudachi.rs

Sudachi in Rust 🦀 and new generation of SudachiPy

What it solves

Sudachi.rs is a Rust implementation of the Sudachi Japanese morphological analyzer. It provides a way to break down Japanese text into meaningful units (tokens) and analyze their grammatical properties, which is essential for processing Japanese language data in high-performance environments.

How it works

The tool uses a system dictionary (available in Small, Core, and Full versions) to perform tokenization and analysis. It supports three different split modes (A, B, and C) to control the granularity of tokenization, ranging from short units to named entities. It also provides normalized forms of words to unify different variations of spellings or expressions.

Who it’s for

Developers and data scientists working with Japanese text processing who need a fast, memory-safe implementation of the Sudachi analyzer in Rust.

Highlights

  • Multi-granular Tokenization: Ability to switch between short (A), middle (B), and named entity (C) split units.
  • Normalized Form: Converts variations of words into a standard form for easier analysis.
  • Wakati Output: Supports space-delimited surface form output for simple tokenization.
  • Dictionary Flexibility: Supports multiple dictionary sizes (Small, Core, Full) and user-defined dictionaries.

Related

  • Project
    WorksApplications/Sudachi

    Sudachi is a Java‑based Japanese morphological analyzer that tokenises text, tags parts of speech, and normalises variants. It offers three segmentation modes, large UniDic‑based dictionaries (small, core, full), a plugin architecture, and support for custom user dictionaries. Use it via a command‑line jar, as a Maven library, or through ports in Python/Rust and an Elasticsearch plugin. Licensed under Apache 2.0.

  • Project
    lindera/lindera

    A high-performance morphological analysis library in Rust for Japanese text tokenization, providing bindings for Python, Node.js, and other languages.

  • Project
    WorksApplications/SudachiDict

    A lexicon for the Sudachi Japanese tokenizer that provides various dictionary sizes for accurate word segmentation and morphological analysis.

  • Project
    huggingface/tokenizers

    A Rust‑based, ultra‑fast tokenizer library (with Python/Node bindings) that implements BPE, WordPiece, Unigram, etc., separates inference from training, uses SIMD‑accelerated pre‑tokenization, and provides a tiny production binary.

  • Project
    polm/cutlet

    A tool for converting Japanese text to romaji with support for multiple romanization systems and URL slug generation.