messense/jieba-rs

The Jieba Chinese Word Segmentation Implemented in Rust

What it solves

It provides a high-performance implementation of the Jieba Chinese word segmentation algorithm in Rust, allowing developers to break down Chinese text into individual words for downstream natural language processing tasks.

How it works

The project implements the Jieba segmentation logic in Rust to achieve better performance. It supports basic word cutting (segmentation) and offers optional features for keyword extraction using TF-IDF and TextRank algorithms.

Who it’s for

Developers who need efficient Chinese text processing and word segmentation in Rust or other languages via available bindings (such as Node.js, Python, and PHP).

Highlights

  • High-performance Rust implementation
  • Supports TF-IDF and TextRank for keyword extraction
  • Includes an embedded dictionary by default
  • Wide range of language bindings available for multiple environments

Related

  • Project
    yanyiwu/nodejieba

    A high-performance Node.js implementation of the Jieba Chinese word segmentation tool, providing efficient text splitting, part-of-speech tagging, and keyword extraction.

  • Project
    yanyiwu/gojieba

    A high-performance Go implementation of the Jieba Chinese word segmentation tool, providing multiple segmentation modes and POS tagging.

  • Project
    go-ego/gse

    A high-performance Go library for multilingual text segmentation and NLP, providing efficient word cutting for languages like Chinese, Japanese, and English.

  • Project
    benbrandt/text-splitter

    A Rust-based text splitting library that breaks long documents into semantically sensible chunks for LLM context windows, supporting plain text, Markdown, and source code.

  • Project
    TW-NLP/ChineseErrorCorrector

    A comprehensive platform for Chinese text correction providing SOTA models, evaluation tools, and data augmentation for spelling and grammatical error correction.