TW-NLP/ChineseErrorCorrector

What it solves

This project provides a comprehensive platform for Chinese text correction, addressing both Chinese Spelling Correction (CSC) and Chinese Grammatical Error Correction (CGEC). It solves the problem of identifying and fixing typos, word misalignments, and complex grammatical errors in Chinese text, outperforming general-purpose models like GPT-4 on specialized benchmarks.

How it works

The platform integrates several components to achieve high-accuracy correction:

• Pre-trained Models: A series of specialized models (e.g., ChineseErrorCorrector4-4B) trained on millions of supervised samples and Chain-of-Thought (COT) data to provide reasoning for corrections.
• Inference Pipeline: Supports deployment via transformers, vLLM for high-throughput batch processing, and modelscope for domestic Chinese mirrors.
• Efficiency Gating: An optional lightweight ELECTRA-based detector can be used as a gate to identify potentially erroneous sentences before calling the larger 4B model, reducing computational costs.
• Data Augmentation: A tool that can synthesize 14 types of grammatical errors to create training data for domain-specific correction models.

Who it’s for

• Researchers: Those studying NLP and text correction who need access to curated paper lists, evaluation tools (Common Errant), and SOTA benchmarks.
• Developers: Engineers looking to integrate high-performance Chinese error correction into applications via API or local deployment.
• ML Engineers: Practitioners who want to fine-tune correction models on their own data using the provided augmentation tools and LLaMA-Factory.

Highlights

• SOTA Performance: The ChineseErrorCorrector4-4B model achieves state-of-the-art results on NACGEC and CSCD benchmarks, surpassing GPT-4.
• Comprehensive Toolset: Combines model weights, evaluation tools, and data augmentation in one repository.
• Transparent Reasoning: The latest models provide not just the corrected text, but also the error type and the reason for the modification.
• Data Resources: Open-sources massive datasets, including 2 million supervised samples and 340k COT samples.