metasequoiaime/chinese-ime-lm

What it solves

Chinese input methods often need to pick the right character when a user types pinyin. This project builds language models that help make that choice more accurate, using both traditional n-gram statistics and modern neural networks. It also provides a shared evaluation set so different approaches can be compared fairly.

How it works

The project has two parallel pipelines that both decide among candidate characters for a given pinyin.

• The n-gram route: it cleans raw text into pure Chinese segments, builds 1/2/3-gram counts with KenLM, and decodes using Viterbi.
• The neural route: it fetches clean, re-distributable corpora (C4 and LCCC), trains a character-level Transformer to rerank candidate lists, and exports small int8 models.

Both pipelines are measured with the same evaluation set (25,119 word-level and 60 sentence-level cases). The evaluation set is carefully designed to catch two subtle measurement pitfalls, turning subjective changes into numeric scores.

The neural models are tiny: 4.5 MB and 25.5 MB. In head-to-head tests against much larger open models (Qwen 0.6B/0.8B) on the same candidate pool, the small ones match or exceed ranking quality, but the differences are not statistically significant. The real advantage is size and latency—the small models run far faster and use far less memory.

Who it's for

This is for developers building Chinese input methods or pinyin-to-character conversion systems, and for researchers studying language models for Asian text entry. It's also a useful reference if you need a small, fast reranker that runs on phones or low-resource devices.

Highlights

• Two complete, comparable approaches: classic n-gram + Viterbi and a modern Transformer reranker.
• A rigorous evaluation set with 25,119 word-level and 60 sentence-level cases, plus documented measurement pitfalls.
• Extremely compact neural models (4.5 MB and 25.5 MB) that rival much larger open-source models in ranking quality while being 140–380× smaller and an order of magnitude faster.
• The project only distributes scripts; corpora and trained weights are not included. Clean, re-licensable corpora are fetched via corpus/fetch.py, and trained weights are released under Apache-2.0 on Hugging Face.
• A Rust reference implementation (reference/) with no unsafe code and no dependencies except serde, making integration easy for any input method.