messense/jieba-rs

The Jieba Chinese Word Segmentation Implemented in Rust

What it solves

Chinese word segmentation is a fundamental requirement for processing Chinese text. This project provides a high-performance implementation of the Jieba algorithm in Rust, allowing developers to split Chinese sentences into individual words efficiently.

How it works

It implements the Jieba Chinese word segmentation logic in Rust. The library provides a core Jieba object that can perform word cutting (segmentation) on input strings. It also supports optional features like an embedded dictionary for ease of use and keyword extraction using TF-IDF and TextRank algorithms.

Who it’s for

Developers who need fast, reliable Chinese text processing and segmentation in Rust, or those using other languages via the provided bindings (NodeJS, PHP, Python, R, WASM, etc.).

Highlights

  • High Performance: Optimized for speed, with benchmarks showing it to be significantly faster than cppjieba.
  • Extensible: Offers optional features for TF-IDF and TextRank keyword extraction.
  • Broad Ecosystem: Available as a base for numerous language bindings including Python, NodeJS, and PHP.
  • Easy Installation: Simple integration via Cargo for Rust developers.

Related

  • Project
    yanyiwu/nodejieba

    A high-performance Node.js implementation of the Jieba Chinese word segmentation tool, providing efficient text splitting, part-of-speech tagging, and keyword extraction.

  • Project
    yanyiwu/gojieba

    A high-performance Go implementation of the Jieba Chinese word segmentation tool, providing multiple segmentation modes and POS tagging.

  • Project
    nieldlr/hanzi

    A Node.js module for Chinese language processing that provides character decomposition, dictionary lookups, and sentence segmentation for language learners.

  • Project
    benbrandt/text-splitter

    A Rust-based text splitting library that breaks long documents into semantically sensible chunks for LLM context windows, supporting plain text, Markdown, and source code.

  • Project
    TW-NLP/ChineseErrorCorrector

    A comprehensive platform for Chinese text correction providing SOTA models, evaluation tools, and data augmentation for spelling and grammatical error correction.