dongrixinyu/JioNLP

What it solves

JioNLP is designed to handle the tedious and time-consuming tasks of Chinese Natural Language Processing (NLP) preprocessing and parsing. It provides a comprehensive set of tools to accelerate development by automating common data cleaning, extraction, and analysis tasks that would otherwise require extensive manual coding.

How it works

The library provides a wide array of specialized functions for different NLP stages:

• Gadgets: Tools for parsing license plates, time semantics, ID cards, and location information, as well as converting between Simplified and Traditional Chinese, and converting characters to Pinyin.
• Data Augmentation: Methods for text augmentation including back-translation, homophone substitution, and NER entity replacement.
• Regular Expression Extraction: Tools to clean text and extract or remove specific patterns like emails, URLs, phone numbers, and IP addresses.
• NER & Classification: Utilities for Named Entity Recognition (NER) tag conversion, dataset splitting, and Naive Bayes frequency analysis for text classification.
• Lexicons: Built-in loaders for stop-words, idioms, and Chinese geographical locations.
• LLM Evaluation: Includes MELLM (Mutual Evaluation of Large Language Models), an automatic evaluation algorithm for LLMs without human supervision.

Who it’s for

It is primarily intended for NLP developers and researchers working with Chinese text data who need efficient, high-accuracy preprocessing and parsing tools.

Highlights

• Extensive Chinese-specific tools: Deep support for Chinese ID cards, license plates, and geographical data.
• LLM Evaluation: Integrated automatic evaluation for Large Language Models.
• esoteric Chinese linguistic tools: Pinyin conversion and character radical/structure analysis.
• Comprehensive data cleaning: One-stop functions for normalizing and removing sensitive or redundant information.