nieldlr/hanzi

HanziJS is a Chinese character and NLP module for Chinese language processing for Node.js

What it solves

HanziJS provides a set of tools for Chinese language processing in Node.js, specifically designed to help language learners explore the structure, meaning, and pronunciation of Chinese characters.

How it works

It acts as a wrapper and processing engine for several linguistic datasets, including CC-CEDICT for definitions, the Junda corpus for character frequency, and Leiden University data for word frequency. It provides a programmatic API to query these datasets and perform algorithmic tasks like character decomposition and sentence segmentation.

Who it’s for

Developers building educational tools for Chinese language learners or applications that require basic Chinese Natural Language Processing (NLP) capabilities.

Highlights

  • Character Decomposition: Breaks characters down into components at three different levels: once, radical, or graphical.
  • Dictionary Integration: Performs definition lookups and searches for compound words using CC-CEDICT.
  • Phonetic Analysis: Computes phonetic regularity to see how a character's pronunciation relates to its components.
  • Text Segmentation: Splits Chinese phrases into individual words using a longest-match lookup.
  • Frequency Data: Provides character and word frequency statistics based on established linguistic corpora.

Related

  • Project
    yanyiwu/nodejieba

    A high-performance Node.js implementation of the Jieba Chinese word segmentation tool, providing efficient text splitting, part-of-speech tagging, and keyword extraction.

  • Project
    yanyiwu/gojieba

    A high-performance Go implementation of the Jieba Chinese word segmentation tool, providing multiple segmentation modes and POS tagging.

  • Project
    dongrixinyu/JioNLP

    A Python library for Chinese NLP preprocessing and parsing, providing tools for data cleaning, entity extraction, and automatic LLM evaluation.

  • Project
    messense/jieba-rs

    A high-performance Rust implementation of the Jieba Chinese word segmentation algorithm for efficient text splitting and keyword extraction.

  • Project
    esbatmop/MNBVC

    A massive, open-source Chinese language corpus project aiming for 253TB of data to provide diverse training material for AI models.